{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MNIST Label Bias.ipynb",
      "version": "0.3.2",
      "provenance": [
        {
          "file_id": "1e-h-2WY0N5jhsCWVnMUXH7dpsk_Jxbce",
          "timestamp": 1566861876837
        },
        {
          "file_id": "1gVCCO279_40uvKQoYAuH9RtkPFX7vQZp",
          "timestamp": 1566340109543
        },
        {
          "file_id": "18_XZnOEysafMPd2UnSXBF9zyRfWwJ_kD",
          "timestamp": 1539805982388
        },
        {
          "file_id": "1N8AxvrF2yj9_letT_Qyiw1kqnHWc2P9z",
          "timestamp": 1527046692702
        },
        {
          "file_id": "1KXesuaJv_taoep0IsgX2dJFwp7cXh8LE",
          "timestamp": 1527038865191
        },
        {
          "file_id": "1ge3kg_iEew2sovhePffrIMJ-3bAG4LNo",
          "timestamp": 1527028222589
        },
        {
          "file_id": "1wvMtCkA-OH9EgoMicaxUkgDL7REL3lEM",
          "timestamp": 1526532367207
        },
        {
          "file_id": "1Edth-uC7VbtP9V9tdGJWTqt5AzCb222q",
          "timestamp": 1525968877287
        },
        {
          "file_id": "1WEQItvRWGU9bSUnK1sEEH4Af9_4TLcPc",
          "timestamp": 1525328745803
        },
        {
          "file_id": "1wDYaDGLBjlPXX5AjYHROq6xCa9vlUZe4",
          "timestamp": 1525328296255
        },
        {
          "file_id": "105Otjs7868UQ-XZKeIGVWfeh8XA1X5e2",
          "timestamp": 1525327243034
        },
        {
          "file_id": "1AmjLjt2PNs05ody3fuAvqYiH5wwPHCIs",
          "timestamp": 1525326926180
        },
        {
          "file_id": "1lGpbHeQq4bE6FlhqzBRrZcNngfBNKbLO",
          "timestamp": 1525217857645
        },
        {
          "file_id": "1zF_aZ1J75eOr--EeCcwCNy06roefqUBn",
          "timestamp": 1525214971469
        },
        {
          "file_id": "1PR-37ZPlNG2LamOXsyT-K9us6iYrcVDv",
          "timestamp": 1525212808685
        },
        {
          "file_id": "1q2FSzKuUcBuvgPGkQtMkEoEIuVLCv0yQ",
          "timestamp": 1525209877057
        },
        {
          "file_id": "1gi5fIQpMkzhS-fWuJThZc77p9SonzwiG",
          "timestamp": 1525202599772
        }
      ],
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "//evaluation/analysis/colab/python:rl_colab",
        "kind": "private"
      }
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    }
  },
  "cells": [
      {
      "cell_type": "markdown",
      "metadata": {
        "id": "AN3Ek2S9IIwK",
        "colab_type": "text"
      },
      "source": [
        "Copyright 2019 Google LLC\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "you may not use this file except in compliance with the License.\n",
        "You may obtain a copy of the License at\n",
        "\n",
        "    https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software\n",
        "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "See the License for the specific language governing permissions and\n",
        "limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I-X3l3nBUe0b",
        "colab_type": "text"
      },
      "source": [
        "## MNIST Simulation\n",
        "\n",
        "We investigate the behavior of our method on a variant of the well-known MNIST task.\n",
        "We take the MNIST dataset under the standard train/test split and then randomly select $20\\%$ of the training data points and change their label to $2$, yielding a biased set of labels. On such a dataset, our method  should be able to find appropriate weights so that training on the weighted dataset roughly corresponds to training on the true labels.\n",
        "To this end, we train a classifier with a demographic-parity-like constraint on the predictions of digit $2$; i.e., we encourage a classifier to predict the digit $2$ at a rate of $10\\%$, the rate appearing in the true labels. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOCVfWcvczhF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf \n",
        "from tensorflow.keras.datasets import mnist\n",
        "import numpy as np\n",
        "import copy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H_f7j4s0Ebz5",
        "colab_type": "text"
      },
      "source": [
        "### Load data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2InFV5FCRUsk",
        "colab_type": "code",
        "outputId": "dde8ff5d-adf7-489a-d89e-2d0c64d61dcd",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566867438804,
          "user_tz": 420,
          "elapsed": 1285,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 66
        }
      },
      "source": [
        "(train_xs, train_ys), (test_xs, test_ys) = mnist.load_data()\n",
        "train_xs = train_xs / 255.\n",
        "test_xs = test_xs / 255.\n",
        "train_xs = train_xs.reshape(-1, 28 * 28)\n",
        "test_xs = test_xs.reshape(-1, 28 * 28)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "11501568/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "otEJZXfXPmS_",
        "colab_type": "code",
        "outputId": "cba0bb36-5314-46e9-e409-1367d44f6520",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566867439159,
          "user_tz": 420,
          "elapsed": 309,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 200
        }
      },
      "source": [
        "print(\"Distribution Before\")\n",
        "for i in range(10):\n",
        "  print np.mean(train_ys == i)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Distribution Before\n",
            "0.09871666666666666\n",
            "0.11236666666666667\n",
            "0.0993\n",
            "0.10218333333333333\n",
            "0.09736666666666667\n",
            "0.09035\n",
            "0.09863333333333334\n",
            "0.10441666666666667\n",
            "0.09751666666666667\n",
            "0.09915\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bVgSnzLXgt2",
        "colab_type": "code",
        "outputId": "939702e7-73ee-408c-e9ea-46b8555e2c9d",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566867439543,
          "user_tz": 420,
          "elapsed": 309,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 200
        }
      },
      "source": [
        "train_ys_corrupted = np.copy(train_ys)\n",
        "np.random.seed(12345)\n",
        "idxs = np.random.choice(range(len(train_ys_corrupted)), size=len(train_ys_corrupted)/5, replace=False)\n",
        "train_ys_corrupted[idxs] = 2\n",
        "print(\"Distribution After\")\n",
        "for i in range(10):\n",
        "  print np.mean(train_ys_corrupted == i)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Distribution After\n",
            "0.07875\n",
            "0.08973333333333333\n",
            "0.2791\n",
            "0.0819\n",
            "0.07831666666666667\n",
            "0.07266666666666667\n",
            "0.07966666666666666\n",
            "0.08318333333333333\n",
            "0.07741666666666666\n",
            "0.07926666666666667\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QSpeUZzI0a1",
        "colab_type": "text"
      },
      "source": [
        "## Neural Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r0iw2bnPI26K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def weight_variable(shape, name=\"weight_variable\"):\n",
        "  \"\"\"weight_variable generates a weight variable of a given shape.\"\"\"\n",
        "  initial = tf.truncated_normal(shape, stddev=0.1)\n",
        "  return tf.Variable(initial, name=name)\n",
        "\n",
        "\n",
        "def bias_variable(shape, name=\"bias_variable\"):\n",
        "  \"\"\"bias_variable generates a bias variable of a given shape.\"\"\"\n",
        "  initial = tf.constant(0.1, shape=shape)\n",
        "  return tf.Variable(initial, name=name)\n",
        "\n",
        "\n",
        "def run_simple_NN(X,\n",
        "                  y,\n",
        "                  X_test,\n",
        "                  y_test,\n",
        "                  weights,\n",
        "                  num_iter=10000,\n",
        "                  learning_rate=0.001,\n",
        "                  batch_size=128,\n",
        "                  display_steps=1000,\n",
        "                  n_layers=1):\n",
        "  n_labels = np.max(y) + 1\n",
        "  n_features = X.shape[1]\n",
        "  weights_ = weights / (1. * np.sum(weights))\n",
        "  x = tf.placeholder(tf.float32, [None, n_features])\n",
        "  y_ = tf.placeholder(tf.float32, [None, n_labels])\n",
        "  \n",
        "  N = 512\n",
        "  \n",
        "  W_1 = weight_variable([784, N])\n",
        "  b_1 = bias_variable([N])\n",
        "\n",
        "  h_1 = tf.nn.relu(tf.matmul(x, W_1) + b_1)\n",
        "\n",
        "  W_2 = weight_variable([N, N])\n",
        "  b_2 = bias_variable([N])\n",
        "\n",
        "  h_2 = tf.nn.relu(tf.matmul(h_1, W_2) + b_2)\n",
        "\n",
        "  W_3 = weight_variable([N, N])\n",
        "  b_3 = bias_variable([N])\n",
        "\n",
        "  h_3 = tf.nn.relu(tf.matmul(h_2, W_3) + b_3)\n",
        "\n",
        "  W_4 = weight_variable([N, 10])\n",
        "  b_4 = bias_variable([10])\n",
        "\n",
        "  NN_logits =tf.nn.softmax(tf.matmul(h_3, W_4) + b_4)\n",
        "\n",
        "  loss = -tf.reduce_mean(tf.reduce_sum(y_ *tf.log(NN_logits+1e-6),1),0)\n",
        "  acc = tf.reduce_mean(\n",
        "      tf.cast(tf.equal(tf.arg_max(NN_logits,1), tf.arg_max(y_,1)), \"float\"))\n",
        "  train_step = tf.train.AdamOptimizer().minimize(loss)\n",
        "  correct_prediction = tf.equal(tf.argmax(NN_logits, 1), tf.argmax(y_, 1))\n",
        "  accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
        "\n",
        "  def one_hot(ns):\n",
        "    return np.eye(n_labels)[ns]\n",
        "\n",
        "  y_onehot = one_hot(y)\n",
        "  y_test_onehot = one_hot(y_test)\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    for i in range(num_iter):\n",
        "      ns = np.random.choice(range(len(X)), size=50, replace=True, p=weights_)\n",
        "      if (i + 1) % display_steps == 0:\n",
        "        train_accuracy = accuracy.eval(feed_dict={x: X, y_: y_onehot})\n",
        "        test_accuracy = accuracy.eval(feed_dict={x: X_test, y_: y_test_onehot})\n",
        "\n",
        "        print(\"step %d, training accuracy %g, test accuracy %g\" %\n",
        "              (i + 1, train_accuracy, test_accuracy))\n",
        "      train_step.run(\n",
        "          feed_dict={x: X[ns, :], y_: y_onehot[ns, :]})\n",
        "\n",
        "    testing_prediction = tf.argmax(NN_logits, 1).eval(feed_dict={x: X_test})\n",
        "    training_prediction = tf.argmax(NN_logits, 1).eval(feed_dict={x: X})\n",
        "    return training_prediction, testing_prediction\n",
        "\n",
        "\n",
        "  \n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOuQuq9awDGE",
        "colab_type": "text"
      },
      "source": [
        "## Training on unbiased dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s3piQvHbaQUH",
        "colab_type": "code",
        "outputId": "87ec1dd7-4714-4c36-f039-dcd5d60f65db",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566867570675,
          "user_tz": 420,
          "elapsed": 130489,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 183
        }
      },
      "source": [
        "weights = np.array([1] * len(train_ys))\n",
        "test_predictions = run_simple_NN(train_xs, train_ys, test_xs, test_ys, weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "step 1000, training accuracy 0.969, test accuracy 0.9647\n",
            "step 2000, training accuracy 0.977817, test accuracy 0.9674\n",
            "step 3000, training accuracy 0.98295, test accuracy 0.9698\n",
            "step 4000, training accuracy 0.986883, test accuracy 0.9738\n",
            "step 5000, training accuracy 0.986067, test accuracy 0.9745\n",
            "step 6000, training accuracy 0.98915, test accuracy 0.9762\n",
            "step 7000, training accuracy 0.991067, test accuracy 0.9775\n",
            "step 8000, training accuracy 0.99195, test accuracy 0.9794\n",
            "step 9000, training accuracy 0.993867, test accuracy 0.9774\n",
            "step 10000, training accuracy 0.992733, test accuracy 0.9786\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RlhSDPDivqS5",
        "colab_type": "text"
      },
      "source": [
        "## Baseline (unconstrained)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ig7I9UacgUO",
        "colab_type": "code",
        "outputId": "aaf7af84-1730-46a6-d9c1-2912c98fb0d1",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566867697039,
          "user_tz": 420,
          "elapsed": 126330,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 183
        }
      },
      "source": [
        "weights = np.array([1] * len(train_ys))\n",
        "test_predictions = run_simple_NN(train_xs, train_ys_corrupted, test_xs, test_ys, weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "step 1000, training accuracy 0.784717, test accuracy 0.9269\n",
            "step 2000, training accuracy 0.799067, test accuracy 0.9415\n",
            "step 3000, training accuracy 0.796417, test accuracy 0.9149\n",
            "step 4000, training accuracy 0.813567, test accuracy 0.945\n",
            "step 5000, training accuracy 0.816767, test accuracy 0.9502\n",
            "step 6000, training accuracy 0.8257, test accuracy 0.9578\n",
            "step 7000, training accuracy 0.826517, test accuracy 0.947\n",
            "step 8000, training accuracy 0.835667, test accuracy 0.938\n",
            "step 9000, training accuracy 0.83125, test accuracy 0.9314\n",
            "step 10000, training accuracy 0.845467, test accuracy 0.9423\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-xh6aWLqvxfb",
        "colab_type": "text"
      },
      "source": [
        "## Our method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MwdtgFL-xojH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def debias_weights(original_labels, protected_attributes, multipliers):\n",
        "  exponents = np.zeros(len(original_labels))\n",
        "  for i, m in enumerate(multipliers):\n",
        "    exponents -= m * protected_attributes[i]\n",
        "  weights = np.exp(exponents)/ (np.exp(exponents) + np.exp(-exponents))\n",
        "  weights = np.where(original_labels == 2, 1 - weights, weights)\n",
        "  return weights"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_Aq5Jb2xwnm",
        "colab_type": "code",
        "outputId": "da3cf50f-16e3-4eb6-83f2-66306ab36091",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1566882946330,
          "user_tz": 420,
          "elapsed": 15248898,
          "user": {
            "displayName": "Heinrich Jiang",
            "photoUrl": "",
            "userId": "02010368581707572492"
          }
        },
        "colab": {
          "height": 1000
        }
      },
      "source": [
        "multipliers = np.zeros(1)\n",
        "learning_rate = 1.\n",
        "n_iters = 100\n",
        "protected_train = [(train_ys_corrupted == 2)]\n",
        "\n",
        "for it in xrange(n_iters):\n",
        "  print(\"Iteration\", it + 1, \"multiplier\", multipliers)\n",
        "  weights = debias_weights(train_ys_corrupted, protected_train, multipliers)\n",
        "  weights = weights / np.sum(weights)\n",
        "  print(\"Weights for 2\", np.sum(weights[np.where(train_ys_corrupted==2)]))\n",
        "  train_prediction, test_predictions = run_simple_NN(train_xs, train_ys_corrupted, test_xs, test_ys, weights)\n",
        "  violation = np.mean(train_prediction == 2) - 0.1\n",
        "  multipliers -= learning_rate * violation\n",
        "  print()\n",
        "  print()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "('Iteration', 1, 'multiplier', array([0.]))\n",
            "('Weights for 2', 0.27910000000000007)\n",
            "step 1000, training accuracy 0.777633, test accuracy 0.9201\n",
            "step 2000, training accuracy 0.803583, test accuracy 0.9507\n",
            "step 3000, training accuracy 0.80525, test accuracy 0.95\n",
            "step 4000, training accuracy 0.813167, test accuracy 0.9505\n",
            "step 5000, training accuracy 0.82015, test accuracy 0.9499\n",
            "step 6000, training accuracy 0.823233, test accuracy 0.9512\n",
            "step 7000, training accuracy 0.827567, test accuracy 0.9415\n",
            "step 8000, training accuracy 0.833467, test accuracy 0.9337\n",
            "step 9000, training accuracy 0.840083, test accuracy 0.9347\n",
            "step 10000, training accuracy 0.8462, test accuracy 0.9407\n",
            "()\n",
            "()\n",
            "('Iteration', 2, 'multiplier', array([-0.05111667]))\n",
            "('Weights for 2', 0.2686755189012749)\n",
            "step 1000, training accuracy 0.762033, test accuracy 0.8753\n",
            "step 2000, training accuracy 0.791167, test accuracy 0.922\n",
            "step 3000, training accuracy 0.809883, test accuracy 0.9596\n",
            "step 4000, training accuracy 0.8132, test accuracy 0.953\n",
            "step 5000, training accuracy 0.816633, test accuracy 0.9616\n",
            "step 6000, training accuracy 0.81895, test accuracy 0.9454\n",
            "step 7000, training accuracy 0.82565, test accuracy 0.9352\n",
            "step 8000, training accuracy 0.825717, test accuracy 0.9109\n",
            "step 9000, training accuracy 0.83415, test accuracy 0.929\n",
            "step 10000, training accuracy 0.83815, test accuracy 0.9069\n",
            "()\n",
            "()\n",
            "('Iteration', 3, 'multiplier', array([-0.13076667]))\n",
            "('Weights for 2', 0.25195311057486536)\n",
            "step 1000, training accuracy 0.79485, test accuracy 0.9498\n",
            "step 2000, training accuracy 0.7918, test accuracy 0.9297\n",
            "step 3000, training accuracy 0.810417, test accuracy 0.9517\n",
            "step 4000, training accuracy 0.811083, test accuracy 0.9398\n",
            "step 5000, training accuracy 0.816233, test accuracy 0.9601\n",
            "step 6000, training accuracy 0.8187, test accuracy 0.9454\n",
            "step 7000, training accuracy 0.826883, test accuracy 0.9531\n",
            "step 8000, training accuracy 0.825533, test accuracy 0.9305\n",
            "step 9000, training accuracy 0.836667, test accuracy 0.9517\n",
            "step 10000, training accuracy 0.835967, test accuracy 0.9291\n",
            "()\n",
            "()\n",
            "('Iteration', 4, 'multiplier', array([-0.1924]))\n",
            "('Weights for 2', 0.23871705910093274)\n",
            "step 1000, training accuracy 0.772583, test accuracy 0.9015\n",
            "step 2000, training accuracy 0.799783, test accuracy 0.9393\n",
            "step 3000, training accuracy 0.8103, test accuracy 0.9564\n",
            "step 4000, training accuracy 0.812933, test accuracy 0.959\n",
            "step 5000, training accuracy 0.818017, test accuracy 0.955\n",
            "step 6000, training accuracy 0.816317, test accuracy 0.9463\n",
            "step 7000, training accuracy 0.82425, test accuracy 0.9387\n",
            "step 8000, training accuracy 0.828083, test accuracy 0.9466\n",
            "step 9000, training accuracy 0.830783, test accuracy 0.9445\n",
            "step 10000, training accuracy 0.835733, test accuracy 0.9352\n",
            "()\n",
            "()\n",
            "('Iteration', 5, 'multiplier', array([-0.23823333]))\n",
            "('Weights for 2', 0.2287687227880643)\n",
            "step 1000, training accuracy 0.791383, test accuracy 0.9431\n",
            "step 2000, training accuracy 0.802983, test accuracy 0.9478\n",
            "step 3000, training accuracy 0.80875, test accuracy 0.9541\n",
            "step 4000, training accuracy 0.8163, test accuracy 0.9591\n",
            "step 5000, training accuracy 0.81965, test accuracy 0.9621\n",
            "step 6000, training accuracy 0.819467, test accuracy 0.9529\n",
            "step 7000, training accuracy 0.824567, test accuracy 0.959\n",
            "step 8000, training accuracy 0.828733, test accuracy 0.9616\n",
            "step 9000, training accuracy 0.83145, test accuracy 0.9568\n",
            "step 10000, training accuracy 0.832133, test accuracy 0.9434\n",
            "()\n",
            "()\n",
            "('Iteration', 6, 'multiplier', array([-0.28435]))\n",
            "('Weights for 2', 0.21871432728640341)\n",
            "step 1000, training accuracy 0.792867, test accuracy 0.9491\n",
            "step 2000, training accuracy 0.788367, test accuracy 0.9201\n",
            "step 3000, training accuracy 0.807317, test accuracy 0.9556\n",
            "step 4000, training accuracy 0.813067, test accuracy 0.9601\n",
            "step 5000, training accuracy 0.81665, test accuracy 0.9675\n",
            "step 6000, training accuracy 0.823267, test accuracy 0.958\n",
            "step 7000, training accuracy 0.8244, test accuracy 0.969\n",
            "step 8000, training accuracy 0.823133, test accuracy 0.9631\n",
            "step 9000, training accuracy 0.833133, test accuracy 0.9473\n",
            "step 10000, training accuracy 0.834767, test accuracy 0.9458\n",
            "()\n",
            "()\n",
            "('Iteration', 7, 'multiplier', array([-0.32421667]))\n",
            "('Weights for 2', 0.21001924646211284)\n",
            "step 1000, training accuracy 0.7906, test accuracy 0.9401\n",
            "step 2000, training accuracy 0.803867, test accuracy 0.9589\n",
            "step 3000, training accuracy 0.81105, test accuracy 0.9611\n",
            "step 4000, training accuracy 0.812117, test accuracy 0.9587\n",
            "step 5000, training accuracy 0.816033, test accuracy 0.9597\n",
            "step 6000, training accuracy 0.819867, test accuracy 0.9605\n",
            "step 7000, training accuracy 0.820317, test accuracy 0.9449\n",
            "step 8000, training accuracy 0.8252, test accuracy 0.958\n",
            "step 9000, training accuracy 0.831167, test accuracy 0.9527\n",
            "step 10000, training accuracy 0.833283, test accuracy 0.9505\n",
            "()\n",
            "()\n",
            "('Iteration', 8, 'multiplier', array([-0.35315]))\n",
            "('Weights for 2', 0.20372344583258353)\n",
            "step 1000, training accuracy 0.782617, test accuracy 0.9131\n",
            "step 2000, training accuracy 0.8037, test accuracy 0.9594\n",
            "step 3000, training accuracy 0.810633, test accuracy 0.958\n",
            "step 4000, training accuracy 0.811067, test accuracy 0.9506\n",
            "step 5000, training accuracy 0.81405, test accuracy 0.9664\n",
            "step 6000, training accuracy 0.819117, test accuracy 0.9607\n",
            "step 7000, training accuracy 0.82235, test accuracy 0.9627\n",
            "step 8000, training accuracy 0.8231, test accuracy 0.963\n",
            "step 9000, training accuracy 0.828883, test accuracy 0.9602\n",
            "step 10000, training accuracy 0.8299, test accuracy 0.9566\n",
            "()\n",
            "()\n",
            "('Iteration', 9, 'multiplier', array([-0.37626667]))\n",
            "('Weights for 2', 0.19871016752279974)\n",
            "step 1000, training accuracy 0.793933, test accuracy 0.9553\n",
            "step 2000, training accuracy 0.804933, test accuracy 0.9623\n",
            "step 3000, training accuracy 0.809083, test accuracy 0.9562\n",
            "step 4000, training accuracy 0.81315, test accuracy 0.9615\n",
            "step 5000, training accuracy 0.81845, test accuracy 0.9671\n",
            "step 6000, training accuracy 0.818617, test accuracy 0.964\n",
            "step 7000, training accuracy 0.823783, test accuracy 0.9609\n",
            "step 8000, training accuracy 0.823367, test accuracy 0.9659\n",
            "step 9000, training accuracy 0.826317, test accuracy 0.9512\n",
            "step 10000, training accuracy 0.828983, test accuracy 0.9579\n",
            "()\n",
            "()\n",
            "('Iteration', 10, 'multiplier', array([-0.40213333]))\n",
            "('Weights for 2', 0.19312523383095268)\n",
            "step 1000, training accuracy 0.789683, test accuracy 0.943\n",
            "step 2000, training accuracy 0.79645, test accuracy 0.9387\n",
            "step 3000, training accuracy 0.810033, test accuracy 0.9593\n",
            "step 4000, training accuracy 0.813367, test accuracy 0.9645\n",
            "step 5000, training accuracy 0.8186, test accuracy 0.9638\n",
            "step 6000, training accuracy 0.81905, test accuracy 0.9688\n",
            "step 7000, training accuracy 0.820017, test accuracy 0.9622\n",
            "step 8000, training accuracy 0.823367, test accuracy 0.9655\n",
            "step 9000, training accuracy 0.82935, test accuracy 0.9559\n",
            "step 10000, training accuracy 0.831917, test accuracy 0.9622\n",
            "()\n",
            "()\n",
            "('Iteration', 11, 'multiplier', array([-0.42728333]))\n",
            "('Weights for 2', 0.1877267086556154)\n",
            "step 1000, training accuracy 0.797033, test accuracy 0.9548\n",
            "step 2000, training accuracy 0.8022, test accuracy 0.9585\n",
            "step 3000, training accuracy 0.80775, test accuracy 0.9547\n",
            "step 4000, training accuracy 0.813333, test accuracy 0.9581\n",
            "step 5000, training accuracy 0.81635, test accuracy 0.9612\n",
            "step 6000, training accuracy 0.819667, test accuracy 0.9676\n",
            "step 7000, training accuracy 0.8223, test accuracy 0.9698\n",
            "step 8000, training accuracy 0.82145, test accuracy 0.9585\n",
            "step 9000, training accuracy 0.8268, test accuracy 0.9635\n",
            "step 10000, training accuracy 0.828283, test accuracy 0.9613\n",
            "()\n",
            "()\n",
            "('Iteration', 12, 'multiplier', array([-0.44766667]))\n",
            "('Weights for 2', 0.18337878820074713)\n",
            "step 1000, training accuracy 0.78835, test accuracy 0.9396\n",
            "step 2000, training accuracy 0.8014, test accuracy 0.9545\n",
            "step 3000, training accuracy 0.80665, test accuracy 0.9583\n",
            "step 4000, training accuracy 0.812767, test accuracy 0.9632\n",
            "step 5000, training accuracy 0.815467, test accuracy 0.9665\n",
            "step 6000, training accuracy 0.817267, test accuracy 0.9632\n",
            "step 7000, training accuracy 0.820367, test accuracy 0.9657\n",
            "step 8000, training accuracy 0.823117, test accuracy 0.9563\n",
            "step 9000, training accuracy 0.826333, test accuracy 0.9658\n",
            "step 10000, training accuracy 0.828683, test accuracy 0.9605\n",
            "()\n",
            "()\n",
            "('Iteration', 13, 'multiplier', array([-0.46696667]))\n",
            "('Weights for 2', 0.17928788031895582)\n",
            "step 1000, training accuracy 0.792767, test accuracy 0.9497\n",
            "step 2000, training accuracy 0.80425, test accuracy 0.9548\n",
            "step 3000, training accuracy 0.810683, test accuracy 0.9606\n",
            "step 4000, training accuracy 0.81155, test accuracy 0.9624\n",
            "step 5000, training accuracy 0.812583, test accuracy 0.9563\n",
            "step 6000, training accuracy 0.815167, test accuracy 0.9694\n",
            "step 7000, training accuracy 0.819517, test accuracy 0.9645\n",
            "step 8000, training accuracy 0.823667, test accuracy 0.9658\n",
            "step 9000, training accuracy 0.82585, test accuracy 0.9648\n",
            "step 10000, training accuracy 0.826567, test accuracy 0.9611\n",
            "()\n",
            "()\n",
            "('Iteration', 14, 'multiplier', array([-0.48738333]))\n",
            "('Weights for 2', 0.17499087903859625)\n",
            "step 1000, training accuracy 0.79395, test accuracy 0.9515\n",
            "step 2000, training accuracy 0.799367, test accuracy 0.9506\n",
            "step 3000, training accuracy 0.809367, test accuracy 0.968\n",
            "step 4000, training accuracy 0.810717, test accuracy 0.9547\n",
            "step 5000, training accuracy 0.816867, test accuracy 0.9675\n",
            "step 6000, training accuracy 0.819417, test accuracy 0.9648\n",
            "step 7000, training accuracy 0.821417, test accuracy 0.9685\n",
            "step 8000, training accuracy 0.8228, test accuracy 0.9669\n",
            "step 9000, training accuracy 0.824283, test accuracy 0.9655\n",
            "step 10000, training accuracy 0.828367, test accuracy 0.9681\n",
            "()\n",
            "()\n",
            "('Iteration', 15, 'multiplier', array([-0.50358333]))\n",
            "('Weights for 2', 0.17160580675710668)\n",
            "step 1000, training accuracy 0.79145, test accuracy 0.9409\n",
            "step 2000, training accuracy 0.8002, test accuracy 0.9533\n",
            "step 3000, training accuracy 0.808217, test accuracy 0.9603\n",
            "step 4000, training accuracy 0.8116, test accuracy 0.968\n",
            "step 5000, training accuracy 0.817433, test accuracy 0.9698\n",
            "step 6000, training accuracy 0.8195, test accuracy 0.9642\n",
            "step 7000, training accuracy 0.82095, test accuracy 0.965\n",
            "step 8000, training accuracy 0.81965, test accuracy 0.9583\n",
            "step 9000, training accuracy 0.822233, test accuracy 0.9686\n",
            "step 10000, training accuracy 0.827917, test accuracy 0.9573\n",
            "()\n",
            "()\n",
            "('Iteration', 16, 'multiplier', array([-0.5262]))\n",
            "('Weights for 2', 0.16691930717912007)\n",
            "step 1000, training accuracy 0.795, test accuracy 0.955\n",
            "step 2000, training accuracy 0.804417, test accuracy 0.9637\n",
            "step 3000, training accuracy 0.812983, test accuracy 0.9679\n",
            "step 4000, training accuracy 0.812083, test accuracy 0.9633\n",
            "step 5000, training accuracy 0.81375, test accuracy 0.9652\n",
            "step 6000, training accuracy 0.814767, test accuracy 0.9666\n",
            "step 7000, training accuracy 0.814733, test accuracy 0.9649\n",
            "step 8000, training accuracy 0.822133, test accuracy 0.9636\n",
            "step 9000, training accuracy 0.82335, test accuracy 0.9653\n",
            "step 10000, training accuracy 0.826617, test accuracy 0.9673\n",
            "()\n",
            "()\n",
            "('Iteration', 17, 'multiplier', array([-0.54068333]))\n",
            "('Weights for 2', 0.16394392777739675)\n",
            "step 1000, training accuracy 0.795783, test accuracy 0.9561\n",
            "step 2000, training accuracy 0.802833, test accuracy 0.9605\n",
            "step 3000, training accuracy 0.80855, test accuracy 0.967\n",
            "step 4000, training accuracy 0.812783, test accuracy 0.9705\n",
            "step 5000, training accuracy 0.81245, test accuracy 0.9601\n",
            "step 6000, training accuracy 0.8156, test accuracy 0.9673\n",
            "step 7000, training accuracy 0.8181, test accuracy 0.9707\n",
            "step 8000, training accuracy 0.819983, test accuracy 0.9645\n",
            "step 9000, training accuracy 0.826083, test accuracy 0.961\n",
            "step 10000, training accuracy 0.827533, test accuracy 0.9675\n",
            "()\n",
            "()\n",
            "('Iteration', 18, 'multiplier', array([-0.55535]))\n",
            "('Weights for 2', 0.16095264910553062)\n",
            "step 1000, training accuracy 0.787967, test accuracy 0.9337\n",
            "step 2000, training accuracy 0.804417, test accuracy 0.9609\n",
            "step 3000, training accuracy 0.811, test accuracy 0.9639\n",
            "step 4000, training accuracy 0.81155, test accuracy 0.9699\n",
            "step 5000, training accuracy 0.816617, test accuracy 0.9648\n",
            "step 6000, training accuracy 0.81845, test accuracy 0.9701\n",
            "step 7000, training accuracy 0.823067, test accuracy 0.9624\n",
            "step 8000, training accuracy 0.82305, test accuracy 0.9671\n",
            "step 9000, training accuracy 0.823417, test accuracy 0.9627\n",
            "step 10000, training accuracy 0.825017, test accuracy 0.9646\n",
            "()\n",
            "()\n",
            "('Iteration', 19, 'multiplier', array([-0.56786667]))\n",
            "('Weights for 2', 0.158417960785824)\n",
            "step 1000, training accuracy 0.793417, test accuracy 0.9497\n",
            "step 2000, training accuracy 0.798433, test accuracy 0.958\n",
            "step 3000, training accuracy 0.810983, test accuracy 0.9678\n",
            "step 4000, training accuracy 0.808867, test accuracy 0.9641\n",
            "step 5000, training accuracy 0.814733, test accuracy 0.9652\n",
            "step 6000, training accuracy 0.819283, test accuracy 0.9651\n",
            "step 7000, training accuracy 0.819883, test accuracy 0.9702\n",
            "step 8000, training accuracy 0.82335, test accuracy 0.969\n",
            "step 9000, training accuracy 0.822217, test accuracy 0.9614\n",
            "step 10000, training accuracy 0.826867, test accuracy 0.9636\n",
            "()\n",
            "()\n",
            "('Iteration', 20, 'multiplier', array([-0.58501667]))\n",
            "('Weights for 2', 0.1549732510316961)\n",
            "step 1000, training accuracy 0.792533, test accuracy 0.9455\n",
            "step 2000, training accuracy 0.802467, test accuracy 0.9517\n",
            "step 3000, training accuracy 0.810367, test accuracy 0.9692\n",
            "step 4000, training accuracy 0.812017, test accuracy 0.9584\n",
            "step 5000, training accuracy 0.8151, test accuracy 0.9656\n",
            "step 6000, training accuracy 0.817233, test accuracy 0.9711\n",
            "step 7000, training accuracy 0.81575, test accuracy 0.9679\n",
            "step 8000, training accuracy 0.821067, test accuracy 0.9711\n",
            "step 9000, training accuracy 0.820867, test accuracy 0.9677\n",
            "step 10000, training accuracy 0.82835, test accuracy 0.9603\n",
            "()\n",
            "()\n",
            "('Iteration', 21, 'multiplier', array([-0.60653333]))\n",
            "('Weights for 2', 0.1506998194354878)\n",
            "step 1000, training accuracy 0.78585, test accuracy 0.9344\n",
            "step 2000, training accuracy 0.801633, test accuracy 0.9489\n",
            "step 3000, training accuracy 0.807133, test accuracy 0.9655\n",
            "step 4000, training accuracy 0.81335, test accuracy 0.9703\n",
            "step 5000, training accuracy 0.81495, test accuracy 0.9706\n",
            "step 6000, training accuracy 0.818217, test accuracy 0.9708\n",
            "step 7000, training accuracy 0.81945, test accuracy 0.9679\n",
            "step 8000, training accuracy 0.822533, test accuracy 0.9753\n",
            "step 9000, training accuracy 0.820383, test accuracy 0.9678\n",
            "step 10000, training accuracy 0.824467, test accuracy 0.9695\n",
            "()\n",
            "()\n",
            "('Iteration', 22, 'multiplier', array([-0.61721667]))\n",
            "('Weights for 2', 0.1485988187661845)\n",
            "step 1000, training accuracy 0.793783, test accuracy 0.9477\n",
            "step 2000, training accuracy 0.804817, test accuracy 0.9596\n",
            "step 3000, training accuracy 0.81025, test accuracy 0.9677\n",
            "step 4000, training accuracy 0.809783, test accuracy 0.9684\n",
            "step 5000, training accuracy 0.816983, test accuracy 0.9711\n",
            "step 6000, training accuracy 0.817833, test accuracy 0.9698\n",
            "step 7000, training accuracy 0.812217, test accuracy 0.9642\n",
            "step 8000, training accuracy 0.817633, test accuracy 0.9683\n",
            "step 9000, training accuracy 0.820367, test accuracy 0.9655\n",
            "step 10000, training accuracy 0.825033, test accuracy 0.9703\n",
            "()\n",
            "()\n",
            "('Iteration', 23, 'multiplier', array([-0.62805]))\n",
            "('Weights for 2', 0.14648289266087877)\n",
            "step 1000, training accuracy 0.7835, test accuracy 0.9257\n",
            "step 2000, training accuracy 0.806067, test accuracy 0.9606\n",
            "step 3000, training accuracy 0.8107, test accuracy 0.9699\n",
            "step 4000, training accuracy 0.812867, test accuracy 0.9708\n",
            "step 5000, training accuracy 0.81565, test accuracy 0.9683\n",
            "step 6000, training accuracy 0.818567, test accuracy 0.9705\n",
            "step 7000, training accuracy 0.819133, test accuracy 0.9651\n",
            "step 8000, training accuracy 0.818383, test accuracy 0.9691\n",
            "step 9000, training accuracy 0.8215, test accuracy 0.9616\n",
            "step 10000, training accuracy 0.8231, test accuracy 0.9701\n",
            "()\n",
            "()\n",
            "('Iteration', 24, 'multiplier', array([-0.6355]))\n",
            "('Weights for 2', 0.14503647928868627)\n",
            "step 1000, training accuracy 0.790417, test accuracy 0.948\n",
            "step 2000, training accuracy 0.80105, test accuracy 0.9521\n",
            "step 3000, training accuracy 0.806817, test accuracy 0.9579\n",
            "step 4000, training accuracy 0.8131, test accuracy 0.9672\n",
            "step 5000, training accuracy 0.8138, test accuracy 0.9654\n",
            "step 6000, training accuracy 0.814, test accuracy 0.9667\n",
            "step 7000, training accuracy 0.82095, test accuracy 0.9698\n",
            "step 8000, training accuracy 0.82035, test accuracy 0.9615\n",
            "step 9000, training accuracy 0.822683, test accuracy 0.9687\n",
            "step 10000, training accuracy 0.825633, test accuracy 0.9633\n",
            "()\n",
            "()\n",
            "('Iteration', 25, 'multiplier', array([-0.65055]))\n",
            "('Weights for 2', 0.14213662778849556)\n",
            "step 1000, training accuracy 0.793967, test accuracy 0.9534\n",
            "step 2000, training accuracy 0.806233, test accuracy 0.9627\n",
            "step 3000, training accuracy 0.810783, test accuracy 0.9665\n",
            "step 4000, training accuracy 0.8125, test accuracy 0.9663\n",
            "step 5000, training accuracy 0.817083, test accuracy 0.9701\n",
            "step 6000, training accuracy 0.817233, test accuracy 0.972\n",
            "step 7000, training accuracy 0.817917, test accuracy 0.9662\n",
            "step 8000, training accuracy 0.818533, test accuracy 0.9636\n",
            "step 9000, training accuracy 0.820633, test accuracy 0.9637\n",
            "step 10000, training accuracy 0.823233, test accuracy 0.9655\n",
            "()\n",
            "()\n",
            "('Iteration', 26, 'multiplier', array([-0.66061667]))\n",
            "('Weights for 2', 0.14021381293826424)\n",
            "step 1000, training accuracy 0.7847, test accuracy 0.936\n",
            "step 2000, training accuracy 0.799017, test accuracy 0.9523\n",
            "step 3000, training accuracy 0.811367, test accuracy 0.9687\n",
            "step 4000, training accuracy 0.81125, test accuracy 0.9669\n",
            "step 5000, training accuracy 0.816083, test accuracy 0.9649\n",
            "step 6000, training accuracy 0.814983, test accuracy 0.9691\n",
            "step 7000, training accuracy 0.818333, test accuracy 0.9679\n",
            "step 8000, training accuracy 0.821917, test accuracy 0.9681\n",
            "step 9000, training accuracy 0.821083, test accuracy 0.9727\n",
            "step 10000, training accuracy 0.825567, test accuracy 0.9676\n",
            "()\n",
            "()\n",
            "('Iteration', 27, 'multiplier', array([-0.67208333]))\n",
            "('Weights for 2', 0.13804039551355346)\n",
            "step 1000, training accuracy 0.791633, test accuracy 0.9483\n",
            "step 2000, training accuracy 0.80535, test accuracy 0.966\n",
            "step 3000, training accuracy 0.806583, test accuracy 0.9664\n",
            "step 4000, training accuracy 0.81275, test accuracy 0.9681\n",
            "step 5000, training accuracy 0.815033, test accuracy 0.9722\n",
            "step 6000, training accuracy 0.815767, test accuracy 0.9695\n",
            "step 7000, training accuracy 0.817617, test accuracy 0.9693\n",
            "step 8000, training accuracy 0.819783, test accuracy 0.9675\n",
            "step 9000, training accuracy 0.819633, test accuracy 0.9696\n",
            "step 10000, training accuracy 0.823933, test accuracy 0.9671\n",
            "()\n",
            "()\n",
            "('Iteration', 28, 'multiplier', array([-0.68186667]))\n",
            "('Weights for 2', 0.13620044265687245)\n",
            "step 1000, training accuracy 0.79095, test accuracy 0.9491\n",
            "step 2000, training accuracy 0.8043, test accuracy 0.9598\n",
            "step 3000, training accuracy 0.809383, test accuracy 0.9632\n",
            "step 4000, training accuracy 0.811533, test accuracy 0.9613\n",
            "step 5000, training accuracy 0.8155, test accuracy 0.9703\n",
            "step 6000, training accuracy 0.816867, test accuracy 0.9678\n",
            "step 7000, training accuracy 0.819283, test accuracy 0.9732\n",
            "step 8000, training accuracy 0.822, test accuracy 0.9704\n",
            "step 9000, training accuracy 0.8206, test accuracy 0.97\n",
            "step 10000, training accuracy 0.82405, test accuracy 0.9692\n",
            "()\n",
            "()\n",
            "('Iteration', 29, 'multiplier', array([-0.69076667]))\n",
            "('Weights for 2', 0.1345383220368331)\n",
            "step 1000, training accuracy 0.788933, test accuracy 0.9476\n",
            "step 2000, training accuracy 0.801417, test accuracy 0.9596\n",
            "step 3000, training accuracy 0.80685, test accuracy 0.9654\n",
            "step 4000, training accuracy 0.812017, test accuracy 0.9651\n",
            "step 5000, training accuracy 0.81485, test accuracy 0.9723\n",
            "step 6000, training accuracy 0.819117, test accuracy 0.9669\n",
            "step 7000, training accuracy 0.81835, test accuracy 0.9728\n",
            "step 8000, training accuracy 0.817483, test accuracy 0.9695\n",
            "step 9000, training accuracy 0.82215, test accuracy 0.9691\n",
            "step 10000, training accuracy 0.822283, test accuracy 0.9682\n",
            "()\n",
            "()\n",
            "('Iteration', 30, 'multiplier', array([-0.70206667]))\n",
            "('Weights for 2', 0.13244429650032596)\n",
            "step 1000, training accuracy 0.794733, test accuracy 0.9594\n",
            "step 2000, training accuracy 0.806433, test accuracy 0.9668\n",
            "step 3000, training accuracy 0.809533, test accuracy 0.969\n",
            "step 4000, training accuracy 0.813733, test accuracy 0.9672\n",
            "step 5000, training accuracy 0.816233, test accuracy 0.9731\n",
            "step 6000, training accuracy 0.817433, test accuracy 0.9698\n",
            "step 7000, training accuracy 0.818933, test accuracy 0.9732\n",
            "step 8000, training accuracy 0.81695, test accuracy 0.9654\n",
            "step 9000, training accuracy 0.821033, test accuracy 0.9698\n",
            "step 10000, training accuracy 0.820417, test accuracy 0.9714\n",
            "()\n",
            "()\n",
            "('Iteration', 31, 'multiplier', array([-0.70725]))\n",
            "('Weights for 2', 0.131489940270346)\n",
            "step 1000, training accuracy 0.796467, test accuracy 0.9586\n",
            "step 2000, training accuracy 0.805683, test accuracy 0.9658\n",
            "step 3000, training accuracy 0.80965, test accuracy 0.9681\n",
            "step 4000, training accuracy 0.814483, test accuracy 0.9667\n",
            "step 5000, training accuracy 0.81395, test accuracy 0.9653\n",
            "step 6000, training accuracy 0.814883, test accuracy 0.9698\n",
            "step 7000, training accuracy 0.819233, test accuracy 0.9716\n",
            "step 8000, training accuracy 0.821517, test accuracy 0.9712\n",
            "step 9000, training accuracy 0.820967, test accuracy 0.9704\n",
            "step 10000, training accuracy 0.81625, test accuracy 0.9653\n",
            "()\n",
            "()\n",
            "('Iteration', 32, 'multiplier', array([-0.71183333]))\n",
            "('Weights for 2', 0.13064932377409094)\n",
            "step 1000, training accuracy 0.793967, test accuracy 0.9545\n",
            "step 2000, training accuracy 0.8053, test accuracy 0.9626\n",
            "step 3000, training accuracy 0.809983, test accuracy 0.9637\n",
            "step 4000, training accuracy 0.813033, test accuracy 0.9708\n",
            "step 5000, training accuracy 0.808, test accuracy 0.9622\n",
            "step 6000, training accuracy 0.819433, test accuracy 0.9743\n",
            "step 7000, training accuracy 0.82025, test accuracy 0.9702\n",
            "step 8000, training accuracy 0.81865, test accuracy 0.9729\n",
            "step 9000, training accuracy 0.82155, test accuracy 0.9738\n",
            "step 10000, training accuracy 0.8211, test accuracy 0.9712\n",
            "()\n",
            "()\n",
            "('Iteration', 33, 'multiplier', array([-0.7183]))\n",
            "('Weights for 2', 0.1294685433777565)\n",
            "step 1000, training accuracy 0.7909, test accuracy 0.9468\n",
            "step 2000, training accuracy 0.80365, test accuracy 0.9627\n",
            "step 3000, training accuracy 0.805633, test accuracy 0.9654\n",
            "step 4000, training accuracy 0.8041, test accuracy 0.9481\n",
            "step 5000, training accuracy 0.81475, test accuracy 0.9684\n",
            "step 6000, training accuracy 0.8143, test accuracy 0.9722\n",
            "step 7000, training accuracy 0.818967, test accuracy 0.9677\n",
            "step 8000, training accuracy 0.819167, test accuracy 0.9701\n",
            "step 9000, training accuracy 0.822167, test accuracy 0.9706\n",
            "step 10000, training accuracy 0.820467, test accuracy 0.9716\n",
            "()\n",
            "()\n",
            "('Iteration', 34, 'multiplier', array([-0.72281667]))\n",
            "('Weights for 2', 0.1286474939275015)\n",
            "step 1000, training accuracy 0.792383, test accuracy 0.9507\n",
            "step 2000, training accuracy 0.804883, test accuracy 0.9647\n",
            "step 3000, training accuracy 0.8105, test accuracy 0.9678\n",
            "step 4000, training accuracy 0.8125, test accuracy 0.969\n",
            "step 5000, training accuracy 0.81505, test accuracy 0.9719\n",
            "step 6000, training accuracy 0.815367, test accuracy 0.969\n",
            "step 7000, training accuracy 0.818067, test accuracy 0.9637\n",
            "step 8000, training accuracy 0.8191, test accuracy 0.9725\n",
            "step 9000, training accuracy 0.82095, test accuracy 0.9657\n",
            "step 10000, training accuracy 0.81975, test accuracy 0.974\n",
            "()\n",
            "()\n",
            "('Iteration', 35, 'multiplier', array([-0.72523333]))\n",
            "('Weights for 2', 0.1282094337239329)\n",
            "step 1000, training accuracy 0.793783, test accuracy 0.9518\n",
            "step 2000, training accuracy 0.805367, test accuracy 0.9625\n",
            "step 3000, training accuracy 0.811467, test accuracy 0.9672\n",
            "step 4000, training accuracy 0.8112, test accuracy 0.9641\n",
            "step 5000, training accuracy 0.814983, test accuracy 0.9675\n",
            "step 6000, training accuracy 0.815567, test accuracy 0.9705\n",
            "step 7000, training accuracy 0.816583, test accuracy 0.9723\n",
            "step 8000, training accuracy 0.81985, test accuracy 0.9737\n",
            "step 9000, training accuracy 0.8226, test accuracy 0.9683\n",
            "step 10000, training accuracy 0.82045, test accuracy 0.9624\n",
            "()\n",
            "()\n",
            "('Iteration', 36, 'multiplier', array([-0.74118333]))\n",
            "('Weights for 2', 0.12534022973839806)\n",
            "step 1000, training accuracy 0.792967, test accuracy 0.9514\n",
            "step 2000, training accuracy 0.805933, test accuracy 0.9654\n",
            "step 3000, training accuracy 0.809033, test accuracy 0.964\n",
            "step 4000, training accuracy 0.81345, test accuracy 0.9682\n",
            "step 5000, training accuracy 0.813933, test accuracy 0.9708\n",
            "step 6000, training accuracy 0.813067, test accuracy 0.9676\n",
            "step 7000, training accuracy 0.818783, test accuracy 0.9722\n",
            "step 8000, training accuracy 0.81655, test accuracy 0.9723\n",
            "step 9000, training accuracy 0.820833, test accuracy 0.9689\n",
            "step 10000, training accuracy 0.8221, test accuracy 0.9619\n",
            "()\n",
            "()\n",
            "('Iteration', 37, 'multiplier', array([-0.75803333]))\n",
            "('Weights for 2', 0.1223511620127376)\n",
            "step 1000, training accuracy 0.794983, test accuracy 0.9514\n",
            "step 2000, training accuracy 0.802633, test accuracy 0.9561\n",
            "step 3000, training accuracy 0.809867, test accuracy 0.9627\n",
            "step 4000, training accuracy 0.808133, test accuracy 0.9613\n",
            "step 5000, training accuracy 0.813283, test accuracy 0.9727\n",
            "step 6000, training accuracy 0.811133, test accuracy 0.9661\n",
            "step 7000, training accuracy 0.815567, test accuracy 0.9697\n",
            "step 8000, training accuracy 0.81985, test accuracy 0.9713\n",
            "step 9000, training accuracy 0.82125, test accuracy 0.9721\n",
            "step 10000, training accuracy 0.822633, test accuracy 0.9676\n",
            "()\n",
            "()\n",
            "('Iteration', 38, 'multiplier', array([-0.77026667]))\n",
            "('Weights for 2', 0.12020853541495277)\n",
            "step 1000, training accuracy 0.795583, test accuracy 0.9536\n",
            "step 2000, training accuracy 0.806233, test accuracy 0.967\n",
            "step 3000, training accuracy 0.807667, test accuracy 0.9639\n",
            "step 4000, training accuracy 0.813633, test accuracy 0.9697\n",
            "step 5000, training accuracy 0.814183, test accuracy 0.9688\n",
            "step 6000, training accuracy 0.81605, test accuracy 0.9724\n",
            "step 7000, training accuracy 0.818167, test accuracy 0.9729\n",
            "step 8000, training accuracy 0.81925, test accuracy 0.9738\n",
            "step 9000, training accuracy 0.8204, test accuracy 0.9737\n",
            "step 10000, training accuracy 0.82055, test accuracy 0.976\n",
            "()\n",
            "()\n",
            "('Iteration', 39, 'multiplier', array([-0.77353333]))\n",
            "('Weights for 2', 0.11964034023030132)\n",
            "step 1000, training accuracy 0.789167, test accuracy 0.9418\n",
            "step 2000, training accuracy 0.8058, test accuracy 0.9643\n",
            "step 3000, training accuracy 0.807417, test accuracy 0.9662\n",
            "step 4000, training accuracy 0.803417, test accuracy 0.9628\n",
            "step 5000, training accuracy 0.812217, test accuracy 0.967\n",
            "step 6000, training accuracy 0.817533, test accuracy 0.972\n",
            "step 7000, training accuracy 0.818633, test accuracy 0.9732\n",
            "step 8000, training accuracy 0.818317, test accuracy 0.9626\n",
            "step 9000, training accuracy 0.8222, test accuracy 0.9699\n",
            "step 10000, training accuracy 0.820167, test accuracy 0.9692\n",
            "()\n",
            "()\n",
            "('Iteration', 40, 'multiplier', array([-0.77831667]))\n",
            "('Weights for 2', 0.11881136224381822)\n",
            "step 1000, training accuracy 0.794117, test accuracy 0.9562\n",
            "step 2000, training accuracy 0.80135, test accuracy 0.9575\n",
            "step 3000, training accuracy 0.808867, test accuracy 0.9664\n",
            "step 4000, training accuracy 0.812567, test accuracy 0.9674\n",
            "step 5000, training accuracy 0.815, test accuracy 0.9716\n",
            "step 6000, training accuracy 0.813567, test accuracy 0.9722\n",
            "step 7000, training accuracy 0.8172, test accuracy 0.9712\n",
            "step 8000, training accuracy 0.816333, test accuracy 0.9681\n",
            "step 9000, training accuracy 0.821717, test accuracy 0.9747\n",
            "step 10000, training accuracy 0.82235, test accuracy 0.9719\n",
            "()\n",
            "()\n",
            "('Iteration', 41, 'multiplier', array([-0.78311667]))\n",
            "('Weights for 2', 0.11798311914951812)\n",
            "step 1000, training accuracy 0.793383, test accuracy 0.9539\n",
            "step 2000, training accuracy 0.806167, test accuracy 0.964\n",
            "step 3000, training accuracy 0.80425, test accuracy 0.9614\n",
            "step 4000, training accuracy 0.8125, test accuracy 0.9708\n",
            "step 5000, training accuracy 0.813017, test accuracy 0.9717\n",
            "step 6000, training accuracy 0.817633, test accuracy 0.9695\n",
            "step 7000, training accuracy 0.815867, test accuracy 0.9735\n",
            "step 8000, training accuracy 0.818567, test accuracy 0.9733\n",
            "step 9000, training accuracy 0.820733, test accuracy 0.9737\n",
            "step 10000, training accuracy 0.822317, test accuracy 0.9702\n",
            "()\n",
            "()\n",
            "('Iteration', 42, 'multiplier', array([-0.79111667]))\n",
            "('Weights for 2', 0.11661081816665303)\n",
            "step 1000, training accuracy 0.788917, test accuracy 0.9321\n",
            "step 2000, training accuracy 0.805833, test accuracy 0.967\n",
            "step 3000, training accuracy 0.80775, test accuracy 0.9628\n",
            "step 4000, training accuracy 0.80815, test accuracy 0.9633\n",
            "step 5000, training accuracy 0.814067, test accuracy 0.9694\n",
            "step 6000, training accuracy 0.816583, test accuracy 0.9731\n",
            "step 7000, training accuracy 0.81785, test accuracy 0.9727\n",
            "step 8000, training accuracy 0.817333, test accuracy 0.9664\n",
            "step 9000, training accuracy 0.8204, test accuracy 0.9706\n",
            "step 10000, training accuracy 0.819933, test accuracy 0.9686\n",
            "()\n",
            "()\n",
            "('Iteration', 43, 'multiplier', array([-0.79583333]))\n",
            "('Weights for 2', 0.11580650089529414)\n",
            "step 1000, training accuracy 0.795583, test accuracy 0.9543\n",
            "step 2000, training accuracy 0.797433, test accuracy 0.959\n",
            "step 3000, training accuracy 0.808483, test accuracy 0.9656\n",
            "step 4000, training accuracy 0.8098, test accuracy 0.9661\n",
            "step 5000, training accuracy 0.81505, test accuracy 0.9734\n",
            "step 6000, training accuracy 0.811683, test accuracy 0.9685\n",
            "step 7000, training accuracy 0.81085, test accuracy 0.967\n",
            "step 8000, training accuracy 0.816933, test accuracy 0.9748\n",
            "step 9000, training accuracy 0.8189, test accuracy 0.9709\n",
            "step 10000, training accuracy 0.818583, test accuracy 0.9733\n",
            "()\n",
            "()\n",
            "('Iteration', 44, 'multiplier', array([-0.79683333]))\n",
            "('Weights for 2', 0.115636430303722)\n",
            "step 1000, training accuracy 0.790933, test accuracy 0.9434\n",
            "step 2000, training accuracy 0.797883, test accuracy 0.9569\n",
            "step 3000, training accuracy 0.805067, test accuracy 0.9621\n",
            "step 4000, training accuracy 0.81255, test accuracy 0.9693\n",
            "step 5000, training accuracy 0.811983, test accuracy 0.9725\n",
            "step 6000, training accuracy 0.814, test accuracy 0.9719\n",
            "step 7000, training accuracy 0.819583, test accuracy 0.9731\n",
            "step 8000, training accuracy 0.817717, test accuracy 0.9738\n",
            "step 9000, training accuracy 0.819867, test accuracy 0.9703\n",
            "step 10000, training accuracy 0.820533, test accuracy 0.9731\n",
            "()\n",
            "()\n",
            "('Iteration', 45, 'multiplier', array([-0.80161667]))\n",
            "('Weights for 2', 0.11482513836617386)\n",
            "step 1000, training accuracy 0.793683, test accuracy 0.9574\n",
            "step 2000, training accuracy 0.80565, test accuracy 0.9667\n",
            "step 3000, training accuracy 0.810983, test accuracy 0.9675\n",
            "step 4000, training accuracy 0.81035, test accuracy 0.9658\n",
            "step 5000, training accuracy 0.811417, test accuracy 0.966\n",
            "step 6000, training accuracy 0.81525, test accuracy 0.9716\n",
            "step 7000, training accuracy 0.8154, test accuracy 0.9726\n",
            "step 8000, training accuracy 0.815917, test accuracy 0.9707\n",
            "step 9000, training accuracy 0.821217, test accuracy 0.9729\n",
            "step 10000, training accuracy 0.819483, test accuracy 0.9725\n",
            "()\n",
            "()\n",
            "('Iteration', 46, 'multiplier', array([-0.80571667]))\n",
            "('Weights for 2', 0.11413266480813113)\n",
            "step 1000, training accuracy 0.78635, test accuracy 0.9466\n",
            "step 2000, training accuracy 0.805117, test accuracy 0.9625\n",
            "step 3000, training accuracy 0.806183, test accuracy 0.9675\n",
            "step 4000, training accuracy 0.811133, test accuracy 0.9692\n",
            "step 5000, training accuracy 0.8116, test accuracy 0.9672\n",
            "step 6000, training accuracy 0.815783, test accuracy 0.9708\n",
            "step 7000, training accuracy 0.8196, test accuracy 0.9687\n",
            "step 8000, training accuracy 0.817583, test accuracy 0.9718\n",
            "step 9000, training accuracy 0.820383, test accuracy 0.9737\n",
            "step 10000, training accuracy 0.819417, test accuracy 0.9717\n",
            "()\n",
            "()\n",
            "('Iteration', 47, 'multiplier', array([-0.80895]))\n",
            "('Weights for 2', 0.11358847366407428)\n",
            "step 1000, training accuracy 0.78855, test accuracy 0.9497\n",
            "step 2000, training accuracy 0.7967, test accuracy 0.9529\n",
            "step 3000, training accuracy 0.806317, test accuracy 0.964\n",
            "step 4000, training accuracy 0.81375, test accuracy 0.9726\n",
            "step 5000, training accuracy 0.81375, test accuracy 0.9737\n",
            "step 6000, training accuracy 0.814567, test accuracy 0.9718\n",
            "step 7000, training accuracy 0.818817, test accuracy 0.9695\n",
            "step 8000, training accuracy 0.818917, test accuracy 0.9737\n",
            "step 9000, training accuracy 0.819683, test accuracy 0.9738\n",
            "step 10000, training accuracy 0.821483, test accuracy 0.9752\n",
            "()\n",
            "()\n",
            "('Iteration', 48, 'multiplier', array([-0.81396667]))\n",
            "('Weights for 2', 0.11274747155700954)\n",
            "step 1000, training accuracy 0.791917, test accuracy 0.9491\n",
            "step 2000, training accuracy 0.8055, test accuracy 0.9663\n",
            "step 3000, training accuracy 0.808233, test accuracy 0.966\n",
            "step 4000, training accuracy 0.8121, test accuracy 0.9655\n",
            "step 5000, training accuracy 0.815217, test accuracy 0.9696\n",
            "step 6000, training accuracy 0.81405, test accuracy 0.9711\n",
            "step 7000, training accuracy 0.81605, test accuracy 0.9682\n",
            "step 8000, training accuracy 0.816083, test accuracy 0.9723\n",
            "step 9000, training accuracy 0.820433, test accuracy 0.9726\n",
            "step 10000, training accuracy 0.821333, test accuracy 0.9717\n",
            "()\n",
            "()\n",
            "('Iteration', 49, 'multiplier', array([-0.8208]))\n",
            "('Weights for 2', 0.11160846749242317)\n",
            "step 1000, training accuracy 0.7924, test accuracy 0.9528\n",
            "step 2000, training accuracy 0.801983, test accuracy 0.9609\n",
            "step 3000, training accuracy 0.807533, test accuracy 0.9621\n",
            "step 4000, training accuracy 0.809367, test accuracy 0.9635\n",
            "step 5000, training accuracy 0.817433, test accuracy 0.9733\n",
            "step 6000, training accuracy 0.813417, test accuracy 0.9699\n",
            "step 7000, training accuracy 0.818433, test accuracy 0.9682\n",
            "step 8000, training accuracy 0.815267, test accuracy 0.9715\n",
            "step 9000, training accuracy 0.819433, test accuracy 0.976\n",
            "step 10000, training accuracy 0.822567, test accuracy 0.9704\n",
            "()\n",
            "()\n",
            "('Iteration', 50, 'multiplier', array([-0.82821667]))\n",
            "('Weights for 2', 0.11038080923703385)\n",
            "step 1000, training accuracy 0.79275, test accuracy 0.955\n",
            "step 2000, training accuracy 0.80395, test accuracy 0.9661\n",
            "step 3000, training accuracy 0.806433, test accuracy 0.9604\n",
            "step 4000, training accuracy 0.81, test accuracy 0.9684\n",
            "step 5000, training accuracy 0.8104, test accuracy 0.9682\n",
            "step 6000, training accuracy 0.814917, test accuracy 0.9708\n",
            "step 7000, training accuracy 0.817383, test accuracy 0.974\n",
            "step 8000, training accuracy 0.8176, test accuracy 0.9724\n",
            "step 9000, training accuracy 0.820617, test accuracy 0.9761\n",
            "step 10000, training accuracy 0.82025, test accuracy 0.9689\n",
            "()\n",
            "()\n",
            "('Iteration', 51, 'multiplier', array([-0.83413333]))\n",
            "('Weights for 2', 0.10940786849840736)\n",
            "step 1000, training accuracy 0.785967, test accuracy 0.9476\n",
            "step 2000, training accuracy 0.804333, test accuracy 0.9648\n",
            "step 3000, training accuracy 0.80955, test accuracy 0.9712\n",
            "step 4000, training accuracy 0.811183, test accuracy 0.9726\n",
            "step 5000, training accuracy 0.807033, test accuracy 0.9672\n",
            "step 6000, training accuracy 0.8152, test accuracy 0.973\n",
            "step 7000, training accuracy 0.8169, test accuracy 0.9749\n",
            "step 8000, training accuracy 0.8185, test accuracy 0.9724\n",
            "step 9000, training accuracy 0.820117, test accuracy 0.9744\n",
            "step 10000, training accuracy 0.81745, test accuracy 0.9713\n",
            "()\n",
            "()\n",
            "('Iteration', 52, 'multiplier', array([-0.83736667]))\n",
            "('Weights for 2', 0.10887859431088946)\n",
            "step 1000, training accuracy 0.79525, test accuracy 0.9531\n",
            "step 2000, training accuracy 0.803983, test accuracy 0.9608\n",
            "step 3000, training accuracy 0.81115, test accuracy 0.9667\n",
            "step 4000, training accuracy 0.812683, test accuracy 0.9662\n",
            "step 5000, training accuracy 0.814217, test accuracy 0.9704\n",
            "step 6000, training accuracy 0.814217, test accuracy 0.9723\n",
            "step 7000, training accuracy 0.816283, test accuracy 0.9743\n",
            "step 8000, training accuracy 0.81915, test accuracy 0.9693\n",
            "step 9000, training accuracy 0.819017, test accuracy 0.9722\n",
            "step 10000, training accuracy 0.818583, test accuracy 0.9721\n",
            "()\n",
            "()\n",
            "('Iteration', 53, 'multiplier', array([-0.84021667]))\n",
            "('Weights for 2', 0.1084134886273645)\n",
            "step 1000, training accuracy 0.79095, test accuracy 0.9551\n",
            "step 2000, training accuracy 0.804683, test accuracy 0.9616\n",
            "step 3000, training accuracy 0.80365, test accuracy 0.9653\n",
            "step 4000, training accuracy 0.811217, test accuracy 0.9663\n",
            "step 5000, training accuracy 0.811467, test accuracy 0.9699\n",
            "step 6000, training accuracy 0.812317, test accuracy 0.9682\n",
            "step 7000, training accuracy 0.818517, test accuracy 0.9727\n",
            "step 8000, training accuracy 0.817117, test accuracy 0.9729\n",
            "step 9000, training accuracy 0.820767, test accuracy 0.9715\n",
            "step 10000, training accuracy 0.8221, test accuracy 0.9688\n",
            "()\n",
            "()\n",
            "('Iteration', 54, 'multiplier', array([-0.84788333]))\n",
            "('Weights for 2', 0.10716894304672939)\n",
            "step 1000, training accuracy 0.787567, test accuracy 0.9435\n",
            "step 2000, training accuracy 0.804783, test accuracy 0.9663\n",
            "step 3000, training accuracy 0.806767, test accuracy 0.9643\n",
            "step 4000, training accuracy 0.81175, test accuracy 0.9663\n",
            "step 5000, training accuracy 0.814633, test accuracy 0.9717\n",
            "step 6000, training accuracy 0.811917, test accuracy 0.9673\n",
            "step 7000, training accuracy 0.81545, test accuracy 0.9713\n",
            "step 8000, training accuracy 0.818967, test accuracy 0.9739\n",
            "step 9000, training accuracy 0.816867, test accuracy 0.9755\n",
            "step 10000, training accuracy 0.818567, test accuracy 0.9685\n",
            "()\n",
            "()\n",
            "('Iteration', 55, 'multiplier', array([-0.85221667]))\n",
            "('Weights for 2', 0.10646978074777855)\n",
            "step 1000, training accuracy 0.795283, test accuracy 0.9543\n",
            "step 2000, training accuracy 0.80595, test accuracy 0.9656\n",
            "step 3000, training accuracy 0.807233, test accuracy 0.9655\n",
            "step 4000, training accuracy 0.8125, test accuracy 0.971\n",
            "step 5000, training accuracy 0.810617, test accuracy 0.9681\n",
            "step 6000, training accuracy 0.814033, test accuracy 0.9719\n",
            "step 7000, training accuracy 0.814633, test accuracy 0.9671\n",
            "step 8000, training accuracy 0.819217, test accuracy 0.9734\n",
            "step 9000, training accuracy 0.818517, test accuracy 0.9721\n",
            "step 10000, training accuracy 0.819683, test accuracy 0.9752\n",
            "()\n",
            "()\n",
            "('Iteration', 56, 'multiplier', array([-0.85606667]))\n",
            "('Weights for 2', 0.10585119891961113)\n",
            "step 1000, training accuracy 0.793283, test accuracy 0.9535\n",
            "step 2000, training accuracy 0.802833, test accuracy 0.9639\n",
            "step 3000, training accuracy 0.807667, test accuracy 0.9683\n",
            "step 4000, training accuracy 0.81365, test accuracy 0.9698\n",
            "step 5000, training accuracy 0.811667, test accuracy 0.9716\n",
            "step 6000, training accuracy 0.81605, test accuracy 0.9746\n",
            "step 7000, training accuracy 0.80785, test accuracy 0.962\n",
            "step 8000, training accuracy 0.818633, test accuracy 0.9752\n",
            "step 9000, training accuracy 0.819067, test accuracy 0.9679\n",
            "step 10000, training accuracy 0.81695, test accuracy 0.9702\n",
            "()\n",
            "()\n",
            "('Iteration', 57, 'multiplier', array([-0.8584]))\n",
            "('Weights for 2', 0.10547749181312434)\n",
            "step 1000, training accuracy 0.7866, test accuracy 0.9503\n",
            "step 2000, training accuracy 0.804967, test accuracy 0.9662\n",
            "step 3000, training accuracy 0.807517, test accuracy 0.9696\n",
            "step 4000, training accuracy 0.810967, test accuracy 0.9704\n",
            "step 5000, training accuracy 0.8133, test accuracy 0.9712\n",
            "step 6000, training accuracy 0.813033, test accuracy 0.9697\n",
            "step 7000, training accuracy 0.816933, test accuracy 0.9723\n",
            "step 8000, training accuracy 0.813567, test accuracy 0.9678\n",
            "step 9000, training accuracy 0.819283, test accuracy 0.9723\n",
            "step 10000, training accuracy 0.816017, test accuracy 0.9654\n",
            "()\n",
            "()\n",
            "('Iteration', 58, 'multiplier', array([-0.86225]))\n",
            "('Weights for 2', 0.10486284276596354)\n",
            "step 1000, training accuracy 0.788667, test accuracy 0.9486\n",
            "step 2000, training accuracy 0.803383, test accuracy 0.9634\n",
            "step 3000, training accuracy 0.8075, test accuracy 0.9701\n",
            "step 4000, training accuracy 0.810967, test accuracy 0.9724\n",
            "step 5000, training accuracy 0.813683, test accuracy 0.9672\n",
            "step 6000, training accuracy 0.8175, test accuracy 0.9725\n",
            "step 7000, training accuracy 0.81675, test accuracy 0.9694\n",
            "step 8000, training accuracy 0.816267, test accuracy 0.9739\n",
            "step 9000, training accuracy 0.82005, test accuracy 0.9749\n",
            "step 10000, training accuracy 0.820183, test accuracy 0.9755\n",
            "()\n",
            "()\n",
            "('Iteration', 59, 'multiplier', array([-0.86508333]))\n",
            "('Weights for 2', 0.10441207057943308)\n",
            "step 1000, training accuracy 0.7941, test accuracy 0.9571\n",
            "step 2000, training accuracy 0.799633, test accuracy 0.96\n",
            "step 3000, training accuracy 0.809, test accuracy 0.9694\n",
            "step 4000, training accuracy 0.810717, test accuracy 0.9631\n",
            "step 5000, training accuracy 0.812317, test accuracy 0.9697\n",
            "step 6000, training accuracy 0.814, test accuracy 0.9706\n",
            "step 7000, training accuracy 0.81615, test accuracy 0.97\n",
            "step 8000, training accuracy 0.815483, test accuracy 0.9749\n",
            "step 9000, training accuracy 0.818317, test accuracy 0.9694\n",
            "step 10000, training accuracy 0.818683, test accuracy 0.9669\n",
            "()\n",
            "()\n",
            "('Iteration', 60, 'multiplier', array([-0.8705]))\n",
            "('Weights for 2', 0.10355400340239063)\n",
            "step 1000, training accuracy 0.793833, test accuracy 0.9552\n",
            "step 2000, training accuracy 0.803383, test accuracy 0.9641\n",
            "step 3000, training accuracy 0.808383, test accuracy 0.9682\n",
            "step 4000, training accuracy 0.811817, test accuracy 0.9697\n",
            "step 5000, training accuracy 0.811867, test accuracy 0.9653\n",
            "step 6000, training accuracy 0.816, test accuracy 0.9709\n",
            "step 7000, training accuracy 0.817367, test accuracy 0.9722\n",
            "step 8000, training accuracy 0.81995, test accuracy 0.9698\n",
            "step 9000, training accuracy 0.820083, test accuracy 0.9746\n",
            "step 10000, training accuracy 0.817283, test accuracy 0.9723\n",
            "()\n",
            "()\n",
            "('Iteration', 61, 'multiplier', array([-0.8728]))\n",
            "('Weights for 2', 0.10319112741628399)\n",
            "step 1000, training accuracy 0.792667, test accuracy 0.9487\n",
            "step 2000, training accuracy 0.806933, test accuracy 0.9638\n",
            "step 3000, training accuracy 0.804917, test accuracy 0.9613\n",
            "step 4000, training accuracy 0.8072, test accuracy 0.9646\n",
            "step 5000, training accuracy 0.8131, test accuracy 0.9701\n",
            "step 6000, training accuracy 0.815017, test accuracy 0.9717\n",
            "step 7000, training accuracy 0.816217, test accuracy 0.9711\n",
            "step 8000, training accuracy 0.816617, test accuracy 0.9708\n",
            "step 9000, training accuracy 0.820333, test accuracy 0.9721\n",
            "step 10000, training accuracy 0.818467, test accuracy 0.9703\n",
            "()\n",
            "()\n",
            "('Iteration', 62, 'multiplier', array([-0.8761]))\n",
            "('Weights for 2', 0.10267201433074287)\n",
            "step 1000, training accuracy 0.794317, test accuracy 0.9528\n",
            "step 2000, training accuracy 0.804433, test accuracy 0.9621\n",
            "step 3000, training accuracy 0.807733, test accuracy 0.9673\n",
            "step 4000, training accuracy 0.807983, test accuracy 0.9635\n",
            "step 5000, training accuracy 0.8114, test accuracy 0.9719\n",
            "step 6000, training accuracy 0.81535, test accuracy 0.968\n",
            "step 7000, training accuracy 0.817317, test accuracy 0.9739\n",
            "step 8000, training accuracy 0.817717, test accuracy 0.9719\n",
            "step 9000, training accuracy 0.819883, test accuracy 0.9716\n",
            "step 10000, training accuracy 0.81895, test accuracy 0.9708\n",
            "()\n",
            "()\n",
            "('Iteration', 63, 'multiplier', array([-0.88025]))\n",
            "('Weights for 2', 0.10202176106187938)\n",
            "step 1000, training accuracy 0.789733, test accuracy 0.9543\n",
            "step 2000, training accuracy 0.80225, test accuracy 0.962\n",
            "step 3000, training accuracy 0.8096, test accuracy 0.9712\n",
            "step 4000, training accuracy 0.813233, test accuracy 0.9702\n",
            "step 5000, training accuracy 0.80965, test accuracy 0.963\n",
            "step 6000, training accuracy 0.8151, test accuracy 0.9716\n",
            "step 7000, training accuracy 0.816033, test accuracy 0.9721\n",
            "step 8000, training accuracy 0.816467, test accuracy 0.9703\n",
            "step 9000, training accuracy 0.81845, test accuracy 0.9718\n",
            "step 10000, training accuracy 0.820667, test accuracy 0.971\n",
            "()\n",
            "()\n",
            "('Iteration', 64, 'multiplier', array([-0.8848]))\n",
            "('Weights for 2', 0.10131212804578267)\n",
            "step 1000, training accuracy 0.793433, test accuracy 0.9536\n",
            "step 2000, training accuracy 0.8045, test accuracy 0.9631\n",
            "step 3000, training accuracy 0.81065, test accuracy 0.9672\n",
            "step 4000, training accuracy 0.8122, test accuracy 0.9718\n",
            "step 5000, training accuracy 0.814383, test accuracy 0.9709\n",
            "step 6000, training accuracy 0.815033, test accuracy 0.9702\n",
            "step 7000, training accuracy 0.811167, test accuracy 0.9671\n",
            "step 8000, training accuracy 0.817333, test accuracy 0.9722\n",
            "step 9000, training accuracy 0.81975, test accuracy 0.973\n",
            "step 10000, training accuracy 0.815233, test accuracy 0.9658\n",
            "()\n",
            "()\n",
            "('Iteration', 65, 'multiplier', array([-0.88535]))\n",
            "('Weights for 2', 0.10122658193145995)\n",
            "step 1000, training accuracy 0.792333, test accuracy 0.9521\n",
            "step 2000, training accuracy 0.803933, test accuracy 0.9668\n",
            "step 3000, training accuracy 0.8107, test accuracy 0.9715\n",
            "step 4000, training accuracy 0.809417, test accuracy 0.9653\n",
            "step 5000, training accuracy 0.814383, test accuracy 0.9714\n",
            "step 6000, training accuracy 0.814283, test accuracy 0.9667\n",
            "step 7000, training accuracy 0.814017, test accuracy 0.9681\n",
            "step 8000, training accuracy 0.8168, test accuracy 0.9694\n",
            "step 9000, training accuracy 0.82045, test accuracy 0.969\n",
            "step 10000, training accuracy 0.818833, test accuracy 0.9735\n",
            "()\n",
            "()\n",
            "('Iteration', 66, 'multiplier', array([-0.88953333]))\n",
            "('Weights for 2', 0.10057756406852068)\n",
            "step 1000, training accuracy 0.791017, test accuracy 0.948\n",
            "step 2000, training accuracy 0.799333, test accuracy 0.959\n",
            "step 3000, training accuracy 0.809233, test accuracy 0.9718\n",
            "step 4000, training accuracy 0.8116, test accuracy 0.9705\n",
            "step 5000, training accuracy 0.811583, test accuracy 0.9688\n",
            "step 6000, training accuracy 0.8134, test accuracy 0.9691\n",
            "step 7000, training accuracy 0.81635, test accuracy 0.976\n",
            "step 8000, training accuracy 0.81645, test accuracy 0.9745\n",
            "step 9000, training accuracy 0.818667, test accuracy 0.9741\n",
            "step 10000, training accuracy 0.820117, test accuracy 0.9738\n",
            "()\n",
            "()\n",
            "('Iteration', 67, 'multiplier', array([-0.89471667]))\n",
            "('Weights for 2', 0.09977745400525019)\n",
            "step 1000, training accuracy 0.789133, test accuracy 0.9504\n",
            "step 2000, training accuracy 0.802583, test accuracy 0.9615\n",
            "step 3000, training accuracy 0.8072, test accuracy 0.9681\n",
            "step 4000, training accuracy 0.810967, test accuracy 0.9712\n",
            "step 5000, training accuracy 0.813833, test accuracy 0.9704\n",
            "step 6000, training accuracy 0.814767, test accuracy 0.9757\n",
            "step 7000, training accuracy 0.813, test accuracy 0.9711\n",
            "step 8000, training accuracy 0.8175, test accuracy 0.9728\n",
            "step 9000, training accuracy 0.8171, test accuracy 0.9717\n",
            "step 10000, training accuracy 0.81855, test accuracy 0.9729\n",
            "()\n",
            "()\n",
            "('Iteration', 68, 'multiplier', array([-0.89656667]))\n",
            "('Weights for 2', 0.09949297091974303)\n",
            "step 1000, training accuracy 0.795517, test accuracy 0.9575\n",
            "step 2000, training accuracy 0.80245, test accuracy 0.9603\n",
            "step 3000, training accuracy 0.805533, test accuracy 0.968\n",
            "step 4000, training accuracy 0.810617, test accuracy 0.9702\n",
            "step 5000, training accuracy 0.8125, test accuracy 0.9688\n",
            "step 6000, training accuracy 0.815433, test accuracy 0.9711\n",
            "step 7000, training accuracy 0.814967, test accuracy 0.9686\n",
            "step 8000, training accuracy 0.81855, test accuracy 0.9727\n",
            "step 9000, training accuracy 0.818183, test accuracy 0.9721\n",
            "step 10000, training accuracy 0.820633, test accuracy 0.9731\n",
            "()\n",
            "()\n",
            "('Iteration', 69, 'multiplier', array([-0.90281667]))\n",
            "('Weights for 2', 0.09853611156786675)\n",
            "step 1000, training accuracy 0.795383, test accuracy 0.9606\n",
            "step 2000, training accuracy 0.80185, test accuracy 0.9663\n",
            "step 3000, training accuracy 0.808167, test accuracy 0.9722\n",
            "step 4000, training accuracy 0.812233, test accuracy 0.9716\n",
            "step 5000, training accuracy 0.81375, test accuracy 0.9707\n",
            "step 6000, training accuracy 0.815983, test accuracy 0.9726\n",
            "step 7000, training accuracy 0.818283, test accuracy 0.9749\n",
            "step 8000, training accuracy 0.817517, test accuracy 0.9701\n",
            "step 9000, training accuracy 0.8164, test accuracy 0.9738\n",
            "step 10000, training accuracy 0.817433, test accuracy 0.9725\n",
            "()\n",
            "()\n",
            "('Iteration', 70, 'multiplier', array([-0.90406667]))\n",
            "('Weights for 2', 0.09834552390053027)\n",
            "step 1000, training accuracy 0.790833, test accuracy 0.9545\n",
            "step 2000, training accuracy 0.800817, test accuracy 0.9619\n",
            "step 3000, training accuracy 0.804333, test accuracy 0.9645\n",
            "step 4000, training accuracy 0.808517, test accuracy 0.9693\n",
            "step 5000, training accuracy 0.812767, test accuracy 0.9715\n",
            "step 6000, training accuracy 0.814067, test accuracy 0.9708\n",
            "step 7000, training accuracy 0.815517, test accuracy 0.9741\n",
            "step 8000, training accuracy 0.815683, test accuracy 0.9714\n",
            "step 9000, training accuracy 0.819467, test accuracy 0.9777\n",
            "step 10000, training accuracy 0.819583, test accuracy 0.9705\n",
            "()\n",
            "()\n",
            "('Iteration', 71, 'multiplier', array([-0.9093]))\n",
            "('Weights for 2', 0.09755043713564027)\n",
            "step 1000, training accuracy 0.793383, test accuracy 0.9538\n",
            "step 2000, training accuracy 0.804567, test accuracy 0.9659\n",
            "step 3000, training accuracy 0.810117, test accuracy 0.9661\n",
            "step 4000, training accuracy 0.809433, test accuracy 0.9707\n",
            "step 5000, training accuracy 0.8153, test accuracy 0.975\n",
            "step 6000, training accuracy 0.813717, test accuracy 0.9708\n",
            "step 7000, training accuracy 0.814933, test accuracy 0.9705\n",
            "step 8000, training accuracy 0.816883, test accuracy 0.9712\n",
            "step 9000, training accuracy 0.817967, test accuracy 0.9737\n",
            "step 10000, training accuracy 0.81865, test accuracy 0.9732\n",
            "()\n",
            "()\n",
            "('Iteration', 72, 'multiplier', array([-0.91166667]))\n",
            "('Weights for 2', 0.09719238199826893)\n",
            "step 1000, training accuracy 0.791783, test accuracy 0.9547\n",
            "step 2000, training accuracy 0.803283, test accuracy 0.9611\n",
            "step 3000, training accuracy 0.807033, test accuracy 0.9658\n",
            "step 4000, training accuracy 0.811683, test accuracy 0.9724\n",
            "step 5000, training accuracy 0.812667, test accuracy 0.9741\n",
            "step 6000, training accuracy 0.812483, test accuracy 0.9736\n",
            "step 7000, training accuracy 0.816267, test accuracy 0.975\n",
            "step 8000, training accuracy 0.817333, test accuracy 0.9761\n",
            "step 9000, training accuracy 0.815967, test accuracy 0.9717\n",
            "step 10000, training accuracy 0.817967, test accuracy 0.9697\n",
            "()\n",
            "()\n",
            "('Iteration', 73, 'multiplier', array([-0.91933333]))\n",
            "('Weights for 2', 0.09603893200979717)\n",
            "step 1000, training accuracy 0.795667, test accuracy 0.9578\n",
            "step 2000, training accuracy 0.803317, test accuracy 0.9628\n",
            "step 3000, training accuracy 0.80485, test accuracy 0.965\n",
            "step 4000, training accuracy 0.812067, test accuracy 0.9684\n",
            "step 5000, training accuracy 0.81455, test accuracy 0.9718\n",
            "step 6000, training accuracy 0.814783, test accuracy 0.9749\n",
            "step 7000, training accuracy 0.8157, test accuracy 0.9717\n",
            "step 8000, training accuracy 0.81605, test accuracy 0.9721\n",
            "step 9000, training accuracy 0.8165, test accuracy 0.972\n",
            "step 10000, training accuracy 0.817733, test accuracy 0.9708\n",
            "()\n",
            "()\n",
            "('Iteration', 74, 'multiplier', array([-0.92686667]))\n",
            "('Weights for 2', 0.09491514555363792)\n",
            "step 1000, training accuracy 0.7964, test accuracy 0.9575\n",
            "step 2000, training accuracy 0.803267, test accuracy 0.9652\n",
            "step 3000, training accuracy 0.809933, test accuracy 0.9692\n",
            "step 4000, training accuracy 0.810717, test accuracy 0.9676\n",
            "step 5000, training accuracy 0.81025, test accuracy 0.9655\n",
            "step 6000, training accuracy 0.81495, test accuracy 0.9704\n",
            "step 7000, training accuracy 0.814183, test accuracy 0.9714\n",
            "step 8000, training accuracy 0.818683, test accuracy 0.9742\n",
            "step 9000, training accuracy 0.819633, test accuracy 0.9727\n",
            "step 10000, training accuracy 0.819683, test accuracy 0.9712\n",
            "()\n",
            "()\n",
            "('Iteration', 75, 'multiplier', array([-0.93065]))\n",
            "('Weights for 2', 0.09435435904464805)\n",
            "step 1000, training accuracy 0.79535, test accuracy 0.9575\n",
            "step 2000, training accuracy 0.797367, test accuracy 0.9549\n",
            "step 3000, training accuracy 0.806383, test accuracy 0.9654\n",
            "step 4000, training accuracy 0.8104, test accuracy 0.9712\n",
            "step 5000, training accuracy 0.806017, test accuracy 0.9622\n",
            "step 6000, training accuracy 0.816567, test accuracy 0.9745\n",
            "step 7000, training accuracy 0.81495, test accuracy 0.9741\n",
            "step 8000, training accuracy 0.811367, test accuracy 0.9658\n",
            "step 9000, training accuracy 0.816917, test accuracy 0.9715\n",
            "step 10000, training accuracy 0.8181, test accuracy 0.9683\n",
            "()\n",
            "()\n",
            "('Iteration', 76, 'multiplier', array([-0.93643333]))\n",
            "('Weights for 2', 0.09350176634953056)\n",
            "step 1000, training accuracy 0.787867, test accuracy 0.9549\n",
            "step 2000, training accuracy 0.805233, test accuracy 0.9674\n",
            "step 3000, training accuracy 0.8064, test accuracy 0.9648\n",
            "step 4000, training accuracy 0.811817, test accuracy 0.9711\n",
            "step 5000, training accuracy 0.815183, test accuracy 0.9705\n",
            "step 6000, training accuracy 0.8155, test accuracy 0.9735\n",
            "step 7000, training accuracy 0.814667, test accuracy 0.9713\n",
            "step 8000, training accuracy 0.81625, test accuracy 0.9735\n",
            "step 9000, training accuracy 0.81755, test accuracy 0.9728\n",
            "step 10000, training accuracy 0.817367, test accuracy 0.9739\n",
            "()\n",
            "()\n",
            "('Iteration', 77, 'multiplier', array([-0.93918333]))\n",
            "('Weights for 2', 0.09309832530204057)\n",
            "step 1000, training accuracy 0.79005, test accuracy 0.948\n",
            "step 2000, training accuracy 0.801067, test accuracy 0.9624\n",
            "step 3000, training accuracy 0.809267, test accuracy 0.9692\n",
            "step 4000, training accuracy 0.80775, test accuracy 0.9691\n",
            "step 5000, training accuracy 0.814383, test accuracy 0.9737\n",
            "step 6000, training accuracy 0.815317, test accuracy 0.9745\n",
            "step 7000, training accuracy 0.815783, test accuracy 0.9762\n",
            "step 8000, training accuracy 0.815533, test accuracy 0.9697\n",
            "step 9000, training accuracy 0.816333, test accuracy 0.9726\n",
            "step 10000, training accuracy 0.81935, test accuracy 0.977\n",
            "()\n",
            "()\n",
            "('Iteration', 78, 'multiplier', array([-0.94065]))\n",
            "('Weights for 2', 0.09288367610205303)\n",
            "step 1000, training accuracy 0.775967, test accuracy 0.9249\n",
            "step 2000, training accuracy 0.80355, test accuracy 0.9651\n",
            "step 3000, training accuracy 0.80415, test accuracy 0.963\n",
            "step 4000, training accuracy 0.807367, test accuracy 0.9672\n",
            "step 5000, training accuracy 0.81515, test accuracy 0.9755\n",
            "step 6000, training accuracy 0.81045, test accuracy 0.9674\n",
            "step 7000, training accuracy 0.812483, test accuracy 0.968\n",
            "step 8000, training accuracy 0.815717, test accuracy 0.9715\n",
            "step 9000, training accuracy 0.819883, test accuracy 0.9746\n",
            "step 10000, training accuracy 0.818867, test accuracy 0.9701\n",
            "()\n",
            "()\n",
            "('Iteration', 79, 'multiplier', array([-0.9448]))\n",
            "('Weights for 2', 0.092278274004432)\n",
            "step 1000, training accuracy 0.793167, test accuracy 0.9565\n",
            "step 2000, training accuracy 0.80335, test accuracy 0.9652\n",
            "step 3000, training accuracy 0.80955, test accuracy 0.9682\n",
            "step 4000, training accuracy 0.811833, test accuracy 0.9724\n",
            "step 5000, training accuracy 0.809783, test accuracy 0.9718\n",
            "step 6000, training accuracy 0.8159, test accuracy 0.9749\n",
            "step 7000, training accuracy 0.814617, test accuracy 0.9723\n",
            "step 8000, training accuracy 0.8186, test accuracy 0.9784\n",
            "step 9000, training accuracy 0.81345, test accuracy 0.971\n",
            "step 10000, training accuracy 0.818033, test accuracy 0.9748\n",
            "()\n",
            "()\n",
            "('Iteration', 80, 'multiplier', array([-0.946]))\n",
            "('Weights for 2', 0.09210375715860349)\n",
            "step 1000, training accuracy 0.793567, test accuracy 0.9585\n",
            "step 2000, training accuracy 0.804383, test accuracy 0.9677\n",
            "step 3000, training accuracy 0.809533, test accuracy 0.9699\n",
            "step 4000, training accuracy 0.813, test accuracy 0.9746\n",
            "step 5000, training accuracy 0.8132, test accuracy 0.9725\n",
            "step 6000, training accuracy 0.815917, test accuracy 0.9733\n",
            "step 7000, training accuracy 0.8157, test accuracy 0.9725\n",
            "step 8000, training accuracy 0.818883, test accuracy 0.9738\n",
            "step 9000, training accuracy 0.817933, test accuracy 0.9748\n",
            "step 10000, training accuracy 0.81585, test accuracy 0.9691\n",
            "()\n",
            "()\n",
            "('Iteration', 81, 'multiplier', array([-0.9464]))\n",
            "('Weights for 2', 0.0920456386272294)\n",
            "step 1000, training accuracy 0.7939, test accuracy 0.9528\n",
            "step 2000, training accuracy 0.805683, test accuracy 0.9676\n",
            "step 3000, training accuracy 0.810083, test accuracy 0.9683\n",
            "step 4000, training accuracy 0.81, test accuracy 0.9705\n",
            "step 5000, training accuracy 0.812733, test accuracy 0.9715\n",
            "step 6000, training accuracy 0.812817, test accuracy 0.9716\n",
            "step 7000, training accuracy 0.815183, test accuracy 0.9758\n",
            "step 8000, training accuracy 0.818283, test accuracy 0.9741\n",
            "step 9000, training accuracy 0.819733, test accuracy 0.9761\n",
            "step 10000, training accuracy 0.819067, test accuracy 0.9728\n",
            "()\n",
            "()\n",
            "('Iteration', 82, 'multiplier', array([-0.94921667]))\n",
            "('Weights for 2', 0.09163714822560642)\n",
            "step 1000, training accuracy 0.790883, test accuracy 0.9508\n",
            "step 2000, training accuracy 0.796567, test accuracy 0.9591\n",
            "step 3000, training accuracy 0.811167, test accuracy 0.9708\n",
            "step 4000, training accuracy 0.809483, test accuracy 0.9701\n",
            "step 5000, training accuracy 0.811567, test accuracy 0.9684\n",
            "step 6000, training accuracy 0.809083, test accuracy 0.9654\n",
            "step 7000, training accuracy 0.81565, test accuracy 0.9726\n",
            "step 8000, training accuracy 0.814167, test accuracy 0.9663\n",
            "step 9000, training accuracy 0.817717, test accuracy 0.9723\n",
            "step 10000, training accuracy 0.816667, test accuracy 0.9716\n",
            "()\n",
            "()\n",
            "('Iteration', 83, 'multiplier', array([-0.95523333]))\n",
            "('Weights for 2', 0.09076903738019841)\n",
            "step 1000, training accuracy 0.792167, test accuracy 0.9603\n",
            "step 2000, training accuracy 0.79725, test accuracy 0.9583\n",
            "step 3000, training accuracy 0.807, test accuracy 0.9661\n",
            "step 4000, training accuracy 0.811433, test accuracy 0.9701\n",
            "step 5000, training accuracy 0.813933, test accuracy 0.9731\n",
            "step 6000, training accuracy 0.81495, test accuracy 0.9701\n",
            "step 7000, training accuracy 0.8164, test accuracy 0.9743\n",
            "step 8000, training accuracy 0.81665, test accuracy 0.9746\n",
            "step 9000, training accuracy 0.818067, test accuracy 0.9742\n",
            "step 10000, training accuracy 0.816583, test accuracy 0.974\n",
            "()\n",
            "()\n",
            "('Iteration', 84, 'multiplier', array([-0.95608333]))\n",
            "('Weights for 2', 0.09064688580643457)\n",
            "step 1000, training accuracy 0.793917, test accuracy 0.9531\n",
            "step 2000, training accuracy 0.80005, test accuracy 0.9587\n",
            "step 3000, training accuracy 0.807817, test accuracy 0.9697\n",
            "step 4000, training accuracy 0.810717, test accuracy 0.9694\n",
            "step 5000, training accuracy 0.807133, test accuracy 0.9645\n",
            "step 6000, training accuracy 0.815183, test accuracy 0.9712\n",
            "step 7000, training accuracy 0.81535, test accuracy 0.9735\n",
            "step 8000, training accuracy 0.81705, test accuracy 0.9733\n",
            "step 9000, training accuracy 0.8155, test accuracy 0.9723\n",
            "step 10000, training accuracy 0.81835, test accuracy 0.9734\n",
            "()\n",
            "()\n",
            "('Iteration', 85, 'multiplier', array([-0.95841667]))\n",
            "('Weights for 2', 0.090312191424133)\n",
            "step 1000, training accuracy 0.790583, test accuracy 0.9484\n",
            "step 2000, training accuracy 0.804917, test accuracy 0.9654\n",
            "step 3000, training accuracy 0.8089, test accuracy 0.9692\n",
            "step 4000, training accuracy 0.809717, test accuracy 0.9691\n",
            "step 5000, training accuracy 0.813217, test accuracy 0.9741\n",
            "step 6000, training accuracy 0.813717, test accuracy 0.9686\n",
            "step 7000, training accuracy 0.816, test accuracy 0.9722\n",
            "step 8000, training accuracy 0.8174, test accuracy 0.9756\n",
            "step 9000, training accuracy 0.816917, test accuracy 0.9673\n",
            "step 10000, training accuracy 0.81675, test accuracy 0.9723\n",
            "()\n",
            "()\n",
            "('Iteration', 86, 'multiplier', array([-0.9601]))\n",
            "('Weights for 2', 0.09007130100034887)\n",
            "step 1000, training accuracy 0.792583, test accuracy 0.9598\n",
            "step 2000, training accuracy 0.800483, test accuracy 0.9615\n",
            "step 3000, training accuracy 0.80925, test accuracy 0.9654\n",
            "step 4000, training accuracy 0.8118, test accuracy 0.9711\n",
            "step 5000, training accuracy 0.812767, test accuracy 0.9743\n",
            "step 6000, training accuracy 0.812683, test accuracy 0.9704\n",
            "step 7000, training accuracy 0.816667, test accuracy 0.9735\n",
            "step 8000, training accuracy 0.817517, test accuracy 0.978\n",
            "step 9000, training accuracy 0.81595, test accuracy 0.9706\n",
            "step 10000, training accuracy 0.82015, test accuracy 0.9734\n",
            "()\n",
            "()\n",
            "('Iteration', 87, 'multiplier', array([-0.96343333]))\n",
            "('Weights for 2', 0.08959569401489734)\n",
            "step 1000, training accuracy 0.79085, test accuracy 0.9512\n",
            "step 2000, training accuracy 0.805383, test accuracy 0.9632\n",
            "step 3000, training accuracy 0.8092, test accuracy 0.9684\n",
            "step 4000, training accuracy 0.811367, test accuracy 0.9702\n",
            "step 5000, training accuracy 0.813567, test accuracy 0.9734\n",
            "step 6000, training accuracy 0.810567, test accuracy 0.969\n",
            "step 7000, training accuracy 0.81445, test accuracy 0.9703\n",
            "step 8000, training accuracy 0.8168, test accuracy 0.975\n",
            "step 9000, training accuracy 0.817883, test accuracy 0.9764\n",
            "step 10000, training accuracy 0.81895, test accuracy 0.9751\n",
            "()\n",
            "()\n",
            "('Iteration', 88, 'multiplier', array([-0.96628333]))\n",
            "('Weights for 2', 0.08919052911452884)\n",
            "step 1000, training accuracy 0.794717, test accuracy 0.9595\n",
            "step 2000, training accuracy 0.804617, test accuracy 0.9668\n",
            "step 3000, training accuracy 0.808783, test accuracy 0.9694\n",
            "step 4000, training accuracy 0.811067, test accuracy 0.9691\n",
            "step 5000, training accuracy 0.813217, test accuracy 0.9718\n",
            "step 6000, training accuracy 0.813683, test accuracy 0.9747\n",
            "step 7000, training accuracy 0.815467, test accuracy 0.9737\n",
            "step 8000, training accuracy 0.814983, test accuracy 0.9718\n",
            "step 9000, training accuracy 0.81445, test accuracy 0.9738\n",
            "step 10000, training accuracy 0.815417, test accuracy 0.9705\n",
            "()\n",
            "()\n",
            "('Iteration', 89, 'multiplier', array([-0.9678]))\n",
            "('Weights for 2', 0.08897547079312743)\n",
            "step 1000, training accuracy 0.789767, test accuracy 0.9486\n",
            "step 2000, training accuracy 0.805567, test accuracy 0.9662\n",
            "step 3000, training accuracy 0.809667, test accuracy 0.9703\n",
            "step 4000, training accuracy 0.807233, test accuracy 0.9647\n",
            "step 5000, training accuracy 0.814117, test accuracy 0.9732\n",
            "step 6000, training accuracy 0.8151, test accuracy 0.9709\n",
            "step 7000, training accuracy 0.815067, test accuracy 0.9665\n",
            "step 8000, training accuracy 0.817, test accuracy 0.9706\n",
            "step 9000, training accuracy 0.816467, test accuracy 0.9735\n",
            "step 10000, training accuracy 0.818633, test accuracy 0.9695\n",
            "()\n",
            "()\n",
            "('Iteration', 90, 'multiplier', array([-0.97303333]))\n",
            "('Weights for 2', 0.08823636487686623)\n",
            "step 1000, training accuracy 0.792133, test accuracy 0.9462\n",
            "step 2000, training accuracy 0.8016, test accuracy 0.9655\n",
            "step 3000, training accuracy 0.804683, test accuracy 0.9624\n",
            "step 4000, training accuracy 0.80705, test accuracy 0.9672\n",
            "step 5000, training accuracy 0.812117, test accuracy 0.9751\n",
            "step 6000, training accuracy 0.815667, test accuracy 0.9741\n",
            "step 7000, training accuracy 0.81685, test accuracy 0.9753\n",
            "step 8000, training accuracy 0.81845, test accuracy 0.9757\n",
            "step 9000, training accuracy 0.81535, test accuracy 0.9739\n",
            "step 10000, training accuracy 0.815283, test accuracy 0.9722\n",
            "()\n",
            "()\n",
            "('Iteration', 91, 'multiplier', array([-0.9736]))\n",
            "('Weights for 2', 0.08815660995701172)\n",
            "step 1000, training accuracy 0.788133, test accuracy 0.9515\n",
            "step 2000, training accuracy 0.8057, test accuracy 0.9675\n",
            "step 3000, training accuracy 0.8075, test accuracy 0.9664\n",
            "step 4000, training accuracy 0.812417, test accuracy 0.9721\n",
            "step 5000, training accuracy 0.812283, test accuracy 0.9739\n",
            "step 6000, training accuracy 0.812817, test accuracy 0.9724\n",
            "step 7000, training accuracy 0.815167, test accuracy 0.9719\n",
            "step 8000, training accuracy 0.8157, test accuracy 0.9723\n",
            "step 9000, training accuracy 0.817533, test accuracy 0.9774\n",
            "step 10000, training accuracy 0.81845, test accuracy 0.9719\n",
            "()\n",
            "()\n",
            "('Iteration', 92, 'multiplier', array([-0.97691667]))\n",
            "('Weights for 2', 0.08769088899006305)\n",
            "step 1000, training accuracy 0.790283, test accuracy 0.9471\n",
            "step 2000, training accuracy 0.802767, test accuracy 0.9611\n",
            "step 3000, training accuracy 0.808567, test accuracy 0.9667\n",
            "step 4000, training accuracy 0.81, test accuracy 0.9679\n",
            "step 5000, training accuracy 0.8079, test accuracy 0.9694\n",
            "step 6000, training accuracy 0.813217, test accuracy 0.9734\n",
            "step 7000, training accuracy 0.8163, test accuracy 0.9744\n",
            "step 8000, training accuracy 0.8105, test accuracy 0.9671\n",
            "step 9000, training accuracy 0.817133, test accuracy 0.9754\n",
            "step 10000, training accuracy 0.8175, test accuracy 0.9746\n",
            "()\n",
            "()\n",
            "('Iteration', 93, 'multiplier', array([-0.97613333]))\n",
            "('Weights for 2', 0.08780071702529198)\n",
            "step 1000, training accuracy 0.790333, test accuracy 0.9514\n",
            "step 2000, training accuracy 0.80375, test accuracy 0.9644\n",
            "step 3000, training accuracy 0.808367, test accuracy 0.969\n",
            "step 4000, training accuracy 0.811683, test accuracy 0.9727\n",
            "step 5000, training accuracy 0.812883, test accuracy 0.971\n",
            "step 6000, training accuracy 0.817033, test accuracy 0.9731\n",
            "step 7000, training accuracy 0.816667, test accuracy 0.9723\n",
            "step 8000, training accuracy 0.816383, test accuracy 0.9733\n",
            "step 9000, training accuracy 0.818483, test accuracy 0.9744\n",
            "step 10000, training accuracy 0.81645, test accuracy 0.971\n",
            "()\n",
            "()\n",
            "('Iteration', 94, 'multiplier', array([-0.97653333]))\n",
            "('Weights for 2', 0.08774462177125712)\n",
            "step 1000, training accuracy 0.791867, test accuracy 0.9554\n",
            "step 2000, training accuracy 0.802233, test accuracy 0.9606\n",
            "step 3000, training accuracy 0.807283, test accuracy 0.9703\n",
            "step 4000, training accuracy 0.809533, test accuracy 0.9707\n",
            "step 5000, training accuracy 0.811667, test accuracy 0.9707\n",
            "step 6000, training accuracy 0.814683, test accuracy 0.9733\n",
            "step 7000, training accuracy 0.816983, test accuracy 0.9773\n",
            "step 8000, training accuracy 0.81515, test accuracy 0.9712\n",
            "step 9000, training accuracy 0.818183, test accuracy 0.9751\n",
            "step 10000, training accuracy 0.8182, test accuracy 0.9746\n",
            "()\n",
            "()\n",
            "('Iteration', 95, 'multiplier', array([-0.97746667]))\n",
            "('Weights for 2', 0.08761383715759068)\n",
            "step 1000, training accuracy 0.7918, test accuracy 0.9474\n",
            "step 2000, training accuracy 0.80475, test accuracy 0.9657\n",
            "step 3000, training accuracy 0.80655, test accuracy 0.9669\n",
            "step 4000, training accuracy 0.8085, test accuracy 0.9697\n",
            "step 5000, training accuracy 0.815083, test accuracy 0.974\n",
            "step 6000, training accuracy 0.813233, test accuracy 0.9664\n",
            "step 7000, training accuracy 0.813, test accuracy 0.9729\n",
            "step 8000, training accuracy 0.813817, test accuracy 0.9683\n",
            "step 9000, training accuracy 0.81605, test accuracy 0.9749\n",
            "step 10000, training accuracy 0.81785, test accuracy 0.974\n",
            "()\n",
            "()\n",
            "('Iteration', 96, 'multiplier', array([-0.97718333]))\n",
            "('Weights for 2', 0.0876535241929503)\n",
            "step 1000, training accuracy 0.795717, test accuracy 0.9598\n",
            "step 2000, training accuracy 0.802667, test accuracy 0.9627\n",
            "step 3000, training accuracy 0.806833, test accuracy 0.9694\n",
            "step 4000, training accuracy 0.8105, test accuracy 0.9731\n",
            "step 5000, training accuracy 0.815467, test accuracy 0.9727\n",
            "step 6000, training accuracy 0.815533, test accuracy 0.9753\n",
            "step 7000, training accuracy 0.81665, test accuracy 0.9738\n",
            "step 8000, training accuracy 0.8177, test accuracy 0.9755\n",
            "step 9000, training accuracy 0.815383, test accuracy 0.9735\n",
            "step 10000, training accuracy 0.8194, test accuracy 0.9727\n",
            "()\n",
            "()\n",
            "('Iteration', 97, 'multiplier', array([-0.98105]))\n",
            "('Weights for 2', 0.08711307402273065)\n",
            "step 1000, training accuracy 0.78185, test accuracy 0.9342\n",
            "step 2000, training accuracy 0.8054, test accuracy 0.9655\n",
            "step 3000, training accuracy 0.8083, test accuracy 0.9701\n",
            "step 4000, training accuracy 0.81125, test accuracy 0.9731\n",
            "step 5000, training accuracy 0.811733, test accuracy 0.9711\n",
            "step 6000, training accuracy 0.8139, test accuracy 0.9709\n",
            "step 7000, training accuracy 0.812883, test accuracy 0.9728\n",
            "step 8000, training accuracy 0.817133, test accuracy 0.9746\n",
            "step 9000, training accuracy 0.818233, test accuracy 0.9757\n",
            "step 10000, training accuracy 0.818783, test accuracy 0.9739\n",
            "()\n",
            "()\n",
            "('Iteration', 98, 'multiplier', array([-0.9843]))\n",
            "('Weights for 2', 0.08666075403708995)\n",
            "step 1000, training accuracy 0.791067, test accuracy 0.9538\n",
            "step 2000, training accuracy 0.8013, test accuracy 0.9664\n",
            "step 3000, training accuracy 0.805133, test accuracy 0.9659\n",
            "step 4000, training accuracy 0.809833, test accuracy 0.9705\n",
            "step 5000, training accuracy 0.812217, test accuracy 0.9713\n",
            "step 6000, training accuracy 0.816333, test accuracy 0.9685\n",
            "step 7000, training accuracy 0.81495, test accuracy 0.9706\n",
            "step 8000, training accuracy 0.812283, test accuracy 0.9697\n",
            "step 9000, training accuracy 0.814867, test accuracy 0.9735\n",
            "step 10000, training accuracy 0.81865, test accuracy 0.9767\n",
            "()\n",
            "()\n",
            "('Iteration', 99, 'multiplier', array([-0.98651667]))\n",
            "('Weights for 2', 0.08635326335736561)\n",
            "step 1000, training accuracy 0.7893, test accuracy 0.9489\n",
            "step 2000, training accuracy 0.804083, test accuracy 0.965\n",
            "step 3000, training accuracy 0.806083, test accuracy 0.967\n",
            "step 4000, training accuracy 0.807, test accuracy 0.9659\n",
            "step 5000, training accuracy 0.813567, test accuracy 0.9738\n",
            "step 6000, training accuracy 0.812783, test accuracy 0.9682\n",
            "step 7000, training accuracy 0.81385, test accuracy 0.9709\n",
            "step 8000, training accuracy 0.814817, test accuracy 0.974\n",
            "step 9000, training accuracy 0.82, test accuracy 0.9752\n",
            "step 10000, training accuracy 0.81945, test accuracy 0.9749\n",
            "()\n",
            "()\n",
            "('Iteration', 100, 'multiplier', array([-0.98988333]))\n",
            "('Weights for 2', 0.085887820809975)\n",
            "step 1000, training accuracy 0.796217, test accuracy 0.9561\n",
            "step 2000, training accuracy 0.799483, test accuracy 0.9625\n",
            "step 3000, training accuracy 0.8043, test accuracy 0.9628\n",
            "step 4000, training accuracy 0.80855, test accuracy 0.9703\n",
            "step 5000, training accuracy 0.81085, test accuracy 0.9695\n",
            "step 6000, training accuracy 0.8127, test accuracy 0.9699\n",
            "step 7000, training accuracy 0.813583, test accuracy 0.9723\n",
            "step 8000, training accuracy 0.817483, test accuracy 0.9746\n",
            "step 9000, training accuracy 0.816017, test accuracy 0.9704\n",
            "step 10000, training accuracy 0.817967, test accuracy 0.9732\n",
            "()\n",
            "()\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGtlFDLOU4W7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}
